If you’re not checking your client’s HTTP headers, you’re not giving them good service.
I’m not talking about the stuff in between the <head> and </head> tags, either. I’m talking about the server response that you get before you get all that nice HTML, or that fancy PDF, or whatever else your client’s website is slinging.
That’s because, well, your client’s website isn’t slinging anything. It’s being slung by a server, and the server’s HTTP response is the first thing a web browser – or a web robot like Google’s crawler – will see. There’s information there you may not be properly reviewing.
Implementing important SEO items in the HTTP headers is more common than you may think, and most of the major SEO tools don’t check them. It’s the number one feature we’d love to see in Screaming Frog, apart from scheduled crawls which we also built a utility to do. HTTP headers are so critical to our SEO efforts that we built a tool to use internally called Head Exam. It checks headers for SEO factors and lets us bulk check a list of URLs so we can audit a whole site.
Head Exam is so important to our SEO efforts that we’re releasing it as a free tool.
Tool updated 4/25/18, current version 1.16b
The canonicals in the above screenshot aren’t being pulled from the web page source. Those canonicals are coming from the HTTP header response. Use this tool to check your clients’ headers right now. You might be surprised at what you’ve missed.
Here’s a scenario – let’s pretend you’re working on a technical audit for a client, and you call out an absence of a canonical tag (which is traditionally found in the source code). You compile your findings in your report and send to your client. The client responds to your recommendation and sets forth with implementing a canonical tag to live in the source code. (Keep in mind, for some clients there might be a cost involved in this implementation.) But what if the HTTP header was already properly displaying a canonical URL request? Google reads this request and may have already honored it. You could be wasting your client’s time and money, and displaying a degree of your own negligence.
Why HTTP headers are absolutely critical for every SEO
Did you know that you can declare canonical URLs in the HTTP header? Did you know that you can declare canonical tags in the HTTP header and not have to declare them in the HTML code?
If not, then I’m glad you’re reading this now before you audit a client’s website and tell them to implement canonicals, and they look at you blankly and say that they have. In the HTTP headers. That you didn’t even check. The bigger the client, the more likely they’ve done this. Like who? Well, whitehouse.gov for example. Just take a look at their HTTP header:
In this snippet of header (pulled from our Head Exam tool), we can see that the Whitehouse is declaring canonicals, shortlinks and even share images in the header!
This is because the Whitehouse.gov site is built with Drupal, which is used by a full 6% of the Quantcast top 10k sites. In fact, Builtwith is tracking 617,090 live top websites using Drupal right now. Drupal puts canonicals in the HTTP header by default.
But don’t take my word for it, because I’m not the only one talking about this. Mike King just recently covered this in his incredible deck, The Technical SEO Renaissance.
OK, but is anyone really putting this stuff in their HTTP headers?
Um, yes. I plopped in a random sampling from Alexa top sites… prepare to have your socks blown off:
Not only do all of these huge websites use canonicals, shortlinks and other SEO elements in the HTTP headers, but two of them have huge glaring mistakes:
- washingtonexaminer.com declares a canonical that’s a 404. Further, the domain newsok.com in the canonical is for The Oklahoman, not the Washinton Examiner. Whaaa… ?
- radaronline.com has a shortlink declared that redirects to WordPress. Not their WordPress, but straight up the WordPress.com homepage. Whaaa… ?
The fact is this: WordPress and Drupal both include a lot of important SEO information in the HTTP headers by default, and together those platforms make up something like 70% of the top sites on the internet according to Quantcast. That makes looking at HTTP headers pretty damn important for you as an SEO.
What else can be hiding in the HTTP header?
Tons of stuff. In addition to canonical tags, some of the most important things for SEOs are:
- HREFLANG tags
- Share image
- Content length
- File type
… and more
Is your client using one or more of these? Have you ever checked their HTTP headers? If and when your client calls you out on this, “I didn’t know to check there” isn’t going to cut it.
5-minute HTTP header audit checklist
Use this checklist, along with our Head Exam tool above, to do a quick 5-minute health check of your clients’ HTTP headers. Click the items to expand and see what you should check if an item applies to your site.
How to use Head Exam
Using head exam is really easy. You have two tabs:
Use the Bulk SEO HTTP header check tab to check HTTP headers for a bulk list of URLs. Use the Single HTTP header examiner tab to take an in-depth look at the full HTTP headers for a URL.
Bulk SEO HTTP header check tab
On this tab, simply paste a list of URLs into Column B, starting at row 6. The tool will pull each URL, split up the HTTP headers, and parse out relevant SEO metrics from the headers into the cells to the right. You can see canonicals, shortlinks, share images, and more.
Be careful not to change any of the other cells, as they will populate automatically.
Single HTTP Header Examiner tab
On this tab, you can see the full header response for a single URL, parsed out with important stuff (like link rel tags) highlighted. Simply enter your URL in cell C4, hit enter, and wait for the magic to happen.
That’s it! Hope you start checking HTTP headers every time now. Any questions on how to better use Head Exam, let me know!