Give Google's Crawlers a Cleaner Path with Screaming Frog

The Screaming Frog SEO Spider is freemium software that will help you diagnose your website to turn up technical errors that might be preventing Google from seeing your content properly. Here's the scenario: You've worked hard on your website. You took the time to pick out a color scheme that would go with your logo and imagery that highlighted the layout of your store. You even sat down and wrote out friendly and informative copy for your website, and you're feeling pretty good about the presentation overall.

But when you go to look at your website stats, you don't see that flood of visitors you were hoping for. Even more importantly, you don't see that flow of revenue you wanted.

What went wrong?

Well, with digital marketing, you need to start with your website and work your way out. One of the best ways to check your website is with Screaming Frog.

How Can You Use Screaming Frog?

Screaming Frog
Screaming Frog

Screaming Frog crawls through your site, much like Google or any other search engines, and it displays loads of information about your content. You can get a list of every page, image, and script on your website. You'll also see the title tag, meta description tag, h1 tag, canonical tag, and several other important SEO factors for each and every page.

The good news is that if you're using a CMS like Wordpress, you won't need a specialist to fix most of the issues you'll see in your report. You can just jump in and fix them yourself.

Here are some quick tips for getting the most out of Screaming Frog:

1. Be sure to filter by HTML. (see image above) While seeing a full list of all of your website files can be useful, you'll need the HTML view to focus on your page titles, descriptions, etc.

Screaming Frog HTML View
Screaming Frog HTML View

2. Keep in mind that the free version is limited to 500 items. If you have a blog with a couple hundred posts and multiple images per post, you won't be able to see all the information on your site. Just look for trends. Are your title tags working? Do you have meta description tags?

3. Scroll to the far right and look at your word count for each post. Google isn't happy with super short pages. You don't need novels for every page, but make sure you don't have too many pieces of content under a few hundred words. In fact, there's plenty to suggest that longer pages are even more helpful.

4. Try out the "In Links" feature. When you're looking at your list of pages in Screaming Frog, it can be difficult to think through the connections between those pages. One way to visualize more effectively is to select one of the pages in the list by simply clicking on the listing and then looking at the bottom of the page. Select the "In Link" tab (see the image below), and you'll get a full list of every page on your website that points to the page you selected. This one feature has helped me eliminate lots of troubleshooting time.

Screaming Frog In Links
Screaming Frog In Links

5. Check your status codes. Do you see the number 404 listed under Status Code for any pages? That means that you have a broken pathway. There's no content at the URL listed. You'll want to redirect that page to another using a plugin like Redirection if you have Wordpress or by contacting your webmaster for a more manual approach.

For a full guide on how to handle all of those different numbers, check out Moz's Status Code Guide.

6. See if all of your pages are actually useful. Take a look at all of your pages and see if you have URL features like "category" and "tag" listed in there. If you have a blog, and you're tag-happy (meaning you like to add three or four tags for every post), you could end up with literally hundreds of tags pages. This isn't just a Wordpress issue. It happens for CMS systems of all types.

What you get with a tag page (especially a poorly created one) is a title that says something to the effect of "Tag: Horseshoes." And beneath that title, you have one post that you wrote a year ago. Is that actually useful to people? It can be if you consolidate your tags and take a structured approach, but all of these thin pages (tags with one post, etc.) can be hurting your value in Google.

Some of My Favorite Features

I recorded this quick video to demo two of my favorite features of the software: "Level" and "SERP Snippet."

If you can't see the above video, see it on YouTube.

Level refers to how far down your content is buried in relation to your homepage. So, if you on the homepage, the level is 0 (no need to click further). If you're on the About page or any of the other top nav items, then the level is 1. The higher the number, the more clicks are required to find the content you want. Level becomes especially important when including your Contact page or any other call-to-action. Make it easy for your visitor to get in touch!

SERP Snippet stands for Search Engine Results Page Snippet. In simple terms, this is the little preview of your content that you see in Google or Bing. The Snippet is a great way to see if you've included the most relevant information for your page in both the title and in the description. While meta description tags don't directly affect your search rank, they can be a big help in improving your click-through rate from search results.

What about You?

Have you used Screaming Frog or another site-crawling tool to see how your site is working? What tools and / or features have you found work best for you?