Why Every SEO Specialist Needs Screaming Frog

Talk to people outside of online marketing about SEO, and just watch their reactions.

They get dazed and confused, perhaps dizzy, their eyes go blank and they look lost. If you give them a flashlight and a map, they would have no clue how to get back.

One of the most common questions we face as an agency: “How do you do SEO? How does that even work?”

To help people get a better grasp of how we do our jobs and get the results that bring you money, we decided to detail the tools we use. Without them, we would be the ones lost. These tools make our jobs possible. They make our jobs easier.

One of the most important tools that we use for SEO is Screaming Frog. The first time I heard the name, I had the thought: “What would that sound like?” The name is inspired by a frog that stood up for itself after being cornered by two cats in Dan Sharp’s backyard.

“Well, the SEO Spider was developed to solve the problems I and many of the industry had when performing onsite technical SEO audits,” he told us recently. “It was basically a pain in the ass to get the data you needed to make SEO recommendations, particularly on larger websites. It was so easy to miss a directive here or there and was really time consuming.

“Historically, I used a combination of Xenu and some in-house tools. But when I started as an SEO consultant under the name ‘Screaming Frog,’ I decided to develop some software to use internally with our own SEO projects. By the time Screaming Frog had grown into a search marketing agency, we decided to release the SEO Spider publically to help others as well.”

Imagine if you’re a quarterback and Peyton Manning is your coach. You still have to go out and perform, but Manning is there to tell you how to do it, what you’re doing wrong and how to fix it. The SEO Spider is Manning. You still have to know what to crawl, what to look for and how to fix it once you get the results. But Screaming Frog allows you to do it and puts you and your agency in the best place to have success and get the results you want and need. What makes it such a great tool is it continues to evolve. The Screaming Frog you used two years ago is not the one you use today.

What is it?

As it says on the SEO Spider page, the Screaming Frog SEO Spider is a small desktop program you can install locally on your PC, Mac or Linux machine.

Once you click on the “Find Out More” button and then there are buttons on the right side of your screen to download the appropriate version for your operating system. There’s also a purchase license button that gives you even more options (and more tools you can use than the free version).

The program “spiders” websites’ links, images, CSS, script and apps from a SEO perspective. It fetches the key onsite elements for SEO, presents them in tabs by type and allows you to filter for common SEO issues. If there are specific data points you want, you can take what you need by exporting them into Excel. It allows you to view, analyze and filter the data you crawl as it’s collected, and you can update it continuously in the program’s user interface.

From a SEO perspective, when you realize how much time and effort Screaming Frog saves you, you will wonder how people did SEO before it hit the scene in 2010. That’s especially true when one of your clients has a medium to large website. Imagine doing a SEO audit for a site like Amazon and the millions of pages it has. It’s impossible to do. This tool allows us to check the whole site, find the redirects, Meta refreshes and duplicate page issues – to name a few.

As noted above, this tool allows to export what you crawl to Excel. The URL, page title, Meta description, headings and anything else you want to optimize is done easily and allows you to use it as a base for SEO recommendations.

As the company says on the site, SEO Spider crawls sites like Googlebot. It obeys allow, disallow directives and wildcard support like Googlebot, but presents its own user-agent (Screaming Frog SEO Spider) that obeys specific directives for robots.txt. It only spiders the subdomain you enter and treats other subdomains as external links. It crawls from directory path onwards. That means if you want to crawl your blog, enter the URL. If you have a more complicated setup with subdomains and subfolders, you can specify both.

Even if you don’t do technical SEO, this tool makes your job possible and you need to use it.

“I don’t think the initial idea was anything new really,” Dan said. “It just seemed to be a natural evolution to what we already had. Xenu is awesome but didn’t do enough, so we built it.”

How we use it

When we bring on a new client, one of the first things we do is a site audit.

The site structure audit will ultimately result in a significant amount of key learnings for us. We complete a checklist review of your website to look for larger indexing, crawl ability and duplication issues. We’ll also identify key metrics where the site needs improvement.

Our site audits include a thorough examination of all the following items:

• Proper canonical usage
• Duplicate content
• Page depth
• Title tag usage
• H1 usage
• Meta content usage
• AJAX/JavaScript usage
• Outbound linking
• Internal link structure
• Directives (next, preview, no-follow)
• Parameter handling
• HTTPS usage
• Site Speed

If you were to do this for Amazon, or any large site, it would be impossible. You would miss something, and potentially something huge. Even if you had 500 people to do the audit, it would take so long, you could potentially lose the client. That doesn’t include entering the results, going over them and then solving them once you get that far.

Screaming Frog allows us to do this and do so instantaneously. Can you snap your fingers? It doesn’t take much longer than that to crawl a site. You get what you need for an audit, what’s wrong, what’s working and what needs fixed after you blink.

The aspects of a site audit listed above are made possible by Screaming Frog.

Let’s say you want to look at title tags. Per Google, a good title tag is 70 characters… ish. Actually, it’s determined by the pixel width of the characters you’re using. What is the pixel width of your title? It’s not easy to know exactly. Sure, you could check in WordPress to see if they will work. Then wait for the page to become indexed and check. If you’re savy you might even build an Excel template to count the characters by their predetermined width or try editing the live search results with inspect element in Chrome but that’s a lot of work! With Screaming Frog, you not only get the character count but the pixels you used. That ensures the title tag will appear, without breaks, in Google searches.

That’s just one example. We will touch more on the details of what you get from Screaming Frog and what it does a little later.

Screaming Frog also allows you to crawl every couple of months to make sure the site still functions and works the way we need it to. Since it stores data for you, you can compare the latest crawls against ones you did in the past. It’s been said before, but without this tool, that’s not possible.

“It’s cool and I am sure someday I will look back and be proud of some of our achievements,” Dan said. “But I am pretty self-critical and spend most of the time finding things we could do much better. I can be quite impatient and always want to keep pushing us forward.”

The one thing Screaming Frog doesn’t do yet is tell you how popular your pages are. Given the success and popularity of it, it’s only a matter of time before it does.

Until that happens, you have to get crawls and combine them with Google Analytics. Jim Seward shows you how to tie the two together and why everyone who does SEO needs to do it. As he says in the piece, when you combine the two data streams into one, it gives you a full overview of your content and how popular it is.

Why should you do this, as Seward asks? “You can get an overall picture of your site and identify any issues that are occurring on popular pages. You can see which pages within your site have no page views at all, or the ones that have very few pageviews. Maybe there are issues on these pages that become immediately apparent when you combine the two datasets.”

Screaming Frog alone is the most used and powerful tool we have (sorry Moz and Google). But when you combine it with Analytics, there are no words for the power you feel.

What do you get with Screaming Frog?

When you crawl a website with Screaming Frog, you get the data but it’s delivered in a specific way. This piece from Kevin Vertommen shows you how to analyze your website with Screaming Frog.

For the most part, data is dissected by specific views. Internal pages, External Pages and Response codes for example.

Response codes are one of the more important info you’ll get about pages on your site.

Many people know the basic response status codes, you’ll be able to see all of the status codes for pages and links on your site:

• 200 – OK
• 301 – Permanent redirect
• 302 – Temporary redirect
• 404 – Not found
• 500 – Server error
• 503 – Unavailable

As Vertommen points out, when you get these codes you have to start by checking all the redirects and verify that you’ve used the correct version. He adds that you have to remember that 301 is a permanent redirect, so it passes authority to the new page. That isn’t the case with a 302 redirect.

The 402 errors occur when the page is no longer available – either it’s been deleted or renamed – without redirecting you to a new page. The easiest way to solve this issue is to implement a 301 redirect to a relevant page.

When you use this crawl, you also can find the broken internal links. He suggests that you click on the URL with a 4xx or 5xxx status code and look at it in the In Links tab in the bottom window. That is where you’ll find all of the pages that link to the broken URL.

Screaming Frog allows you to analyze the URLs of a website via the URL tab.

You start by checking the length of each URL by using the length tab. This allows you to spot the unnaturally long URLs. An ideal URL is short, four or five words as Vertommen suggests, and is descriptive. You also need to check for filters on ASCII characters, underscores, uppercase letters, duplicate and dynamic URLs. As Vertommen said, these elements can all cause indexation problems, so it’s best to solve them right away when you uncover the issues.

Page titles are very important for SEO and how well a site does. Every page should have a unique title and the most important keywords are used. As mentioned earlier, an ideal title tag is 60-70 characters. But to get the best indicator if it will work, Screaming Frog measures pixels and the number of them used in those characters. Google doesn’t like titles longer than 512 pixels. In some instances, a title shorter than 60 characters might still be too long. That’s why the pixel count is so huge and why it’s so useful.

Did you know that having a Title over 512 pixels is a factor Google considered when deciding to re-write your title tags? This is a major frustration for SEO’s who’ve carefully written and focused on writing a title tag for click through rate performance only to have Google replace it entirely. One great method to avoid that is to ensure you’re pixel width is smaller than 512

Meta descriptions are the text under the page titles in searches. When you do a search, you have the title and that text underneath is the Meta description. The ideal description for the page is 140 characters and is something taken from the story or page that also sells the story. Think of it like a subhead or summary graph in a newspaper, which are tools to get you to keep reading. It’s another way to sell the searcher to click on your link. You never want to leave these empty if you want to direct traffic to the page. Each Meta description should be unique to each page as well.

As with page titles, there are ways to count the characters, but Screaming Frog allows you to count the pixels. It also has a new feature that allows you to simulate a search snippet and analyze what you see. Whether a desktop, mobile or tablet, you see how it looks and if you need to fix anything. To do this, click on the URL and select SERP snippet at the bottom of your screen.

Make sure your Meta description is somewhere on the page. Imagine being told your favorite actor will be at your favorite bar. When you show up, they’re not there. When your descriptions are on the page, it improves the user experience and trust they give you.

One of the most important aspects of a website is images. In the same refrain, speed of a website is crucial for users and search engines. As this story from Moz points out, the speed of your site actually affects search ranking.

What do the two have to do with each other? The larger the image, the slower your site will run and load because of the bandwidth needed for those images. As Vertommen notes in his piece, it’s best to keep them under 100kb.

What Screaming Frog allows you to do is crawl all the images on your site, view the images over 100kb via the filter tab and export the images that are too big.

Screaming Frog also allows you to see the response time of your site and pages under the response codes tab. If there are pages you see that don’t have a good response, use tools such as Webpagetest.org and PageSpeed Insights. Both allow you to dig deeper into the problem and find the solution.

With the Directives tab, this is where you get the information about meta robots, canonical links and rel=next/prev annotations. Screaming Frog allows you to filter the specific directive you want to look for.

If that wasn’t enough, Screaming Frog generates a XML Sitemap. According to Google, a sitemap is a file of where you can list web pages of your site to tell Google and other search engines about the organization of your site content. Search engine web crawlers like Googlebot read this file to intelligently crawl your site.

A sitemap is exactly how it sounds. It’s a map to help the Googlebot crawl your site and make sure it finds all of the relevant content. Screaming Frog has this feature in the top navigation bar under Sitemaps > Create XML Sitemap. Once the file has been created, you can modify the priority and change the frequency of pages.

With all that said, if you’re taking advantage of all the crawling savvy Screaming Frog has to offer, you’ll likely find that a sitemap is not required. You can simply identify and simplify the crawling of your website to the point where a Sitemap is no longer important. Besides, no one likes error checking sitemaps in Google Webmaster Tools, and now you won’t have to!

Screaming Frog is Huge

If there is a SEO who doesn’t use this tool, run away. That should be one of the questions potential clients ask prospective agencies. “Do you use Screaming Frog? Oh, you don’t. Well, have a nice day.”

The program offers a free option, so there’s really no excuse to not use it.

“We had absolutely no expectations really,” Dan said. “There was no big plan. I knew we had built something awesome, but really didn’t expect the SEO community to embrace it quite like they have.

“I remember being over the moon at the first person liking it enough to buy a license, particularly as it was someone I respected so much in the industry. Thanks, Branko.”

As for the future, there is no end in sight to the power and popularity of Screaming Frog. As noted earlier, there are ways to make it even better, like adding a way to measure page popularity. Given that Dan and the company are always looking for ways to do things better, it’s only a matter of time before they find new ways to make our lives as SEOs even easier.

“There is a lot more to come from the SEO Spider and us as an agency,” Dan said. “But we also have new tools that we might release in the future, too. The Year 2015 should be a really exciting year.”