We recently had a client launch a new site in WordPress. It was appropriately in a staging area before launch. Instinctively upon hearing the news of the launch, we decided to look for a robots <meta> tag. Sure enough, every page was marked as “noindex, nofollow”. The client was able to make the change before Google crawled the new site. A major SEO issue averted!
Above I said “instinctively” because, well, this isn’t the first time I’ve seen a site launch set to block search engines. It’s probably not even the 50th. I worked on an eCommerce platform where many sites launched with this SEO issue. WordPress – as fine a platform as it is – makes it super easy to launch set to noindex. Since developers often build sites in staging areas, they’re wise to block bots from inadvertently discovering their playground. But, in the hustle to push live an update or new design, they can forget a tiny (yet crucial) check box.
I’ve gathered up three different ways you can monitor your sites for SEO problems without the use of server logs or an education in server administration. There’s different kinds of advanced website monitoring (e.g., active, passive), but I’m keeping it simple and applicable for anyone. I wanted to pick a few that were diverse, free or affordable.
SEOradar is a great tool from my SEO friend and speaker Mark Munroe. The purpose of the tool is to serve as a type of Google Alerts for when technical SEO changes occur. You can set monitoring to be daily or longer, and cover things like changes in robots.txt, meta robots, internal links, schema, you name it! Plus, you can get these alerts emailed to you.
From the website’s homepage: “SEORadar examines changes to pages and alerts users to issues with potential dire SEO consequences; including title changes, noindex tags, broken canonicals, 302s and much more. SEORadar checks for over 100 distinct site changes and warns users when they occur. Users can configure their own tests if we missed ones they need!”
Here’s an example of the dashboard for one of our enterprise clients. All is mostly calm today with only 2 minor changes to pay attention to. When they’re critical, we jump:
The interface is smart, allowing you to drill deeper into the alerts for identifying the SEO issue. But one feature I really like is being able to view two different source code documents together with the changes highlighted:
2. Uptime Robot
While SEOradar can track a lot of technical SEO related details, the biggest problem your business faces is website downtime. I’m sure we’ve all been there – it’s 2pm in the afternoon, you’re relaxed after lunch when somebody finally notices the site is down. You do some digging, and it turns out the site has been down since 4am. Most companies don’t think to look at their own website routinely. We all just assume it’s alive and well. All webhosts suffer from periodic downtime, but poor hosts – even in this day and age – can really screw you on this one.
Website monitoring services exist to keep you in the loop, from SaaS services like Pingdom and SiteUptime, to downloadable applications. I’m particularly fond of Uptime Robot because it’s easily configurable and free for 50 monitors. If your site goes down, not only do you get an email alert, but you’ve got a great interface to follow along:
Plus you can change the intervals, starting at 5 minutes. Keeps the control in your hands:
3. Daily Rank Tracking and Analytics
If you’re old-school and don’t want more tools in your arsenal, this one is for you.
I was once pitched by one of those enterprise level all-in-one SEO solutions. When they discussed their rank tracking capability, I asked, “can you track rankings on a daily basis?” The smug response: “no, we do weekly rank updates – but I can’t see a reason why you would ever want daily.” Well, on very large sites where popular pages are crawled every couple days, and many updates are being made, daily rank tracking can help you quickly track down where an issue lies. It can usually lead you directly to the page that has the unexpected problem. Not to mention, if you’ve been stung by a penalty, daily rank tracking (and analytics) will often be your first indicator. I like to know that day… not in a week.
Analytics isn’t any different. Whether you’re using free Google Analytics or whatever Adobe is calling their Omniture these days, the data pours in fast. In less than 24 hours, most sites will be able to provide enough clues that a problem is occuring. You just have to monitor routinely.
Daily Rank Tracking
Quick disclaimer: Honestly this isn’t my favorite method of monitoring for SEO problems, but it’s an easy one. If you have a page Google favors, it will get crawled often – as often as daily. If it’s a deeper, lost page, I’ve seen it take months to be revisited. So, if you’re going to rely on rank tracking, choose daily tracking, and monitor your oft-visited pages. Clearly this leaves a lot of monitoring on the floor – be aware, there’s plenty of room for error and latency.
But let’s see show how fast Google is with the following mini experiment. It turns out our site has a crappy local landing page from years ago that I completely forgot about (thanks to SEMrush data for showing me this forgotten page). This page ranks 10th for “lehigh valley SEO”. Lehigh Valley is an area close to our office to which we wanted a local SEO presence. However, it’s last cache date is April 19 (today is April 20). This is an orphaned page except for my sitemap, so it’s not likely to be visited quickly. My next move is to completely sabotaged this page but putting a noindex tag on it. I want this cheesy page gone, not just for this experiment, but because these kinds of pages are more sketchy than ever.
Updated on 5/1/2015: Google finally returned and deindexed the page on 4/27. How do I know? Daily rank tracking. Though it was anything but immediate, it does alert me to review the page for problems. Granted, I intentionally sabotaged this page, but if I hadn’t, this would have highlighted the problem.
Most analytics packages allow you to create a custom dashboard. Most also allow for daily emails. To make this work, your new job is to simply check your analytics a few times a day. For over a decade, this was the first thing I’d do when I got into the office. I wasn’t just looking at revenue, but hiccups. I’m looking for an atypical pattern.
When Google added Real-Time reports, I was thrilled (I had been using Woopra prior). I’ll often have a tab open on a monitor just letting the live data fly:
It’s almost like a heartbeat monitor in a hospital. Clearly this is not as simple as allowing SEOradar or Uptime Robot to send you an alert, but useful nonetheless. What do you use to monitor your website?