Screaming Frog SEO Spider Tool – A Complete Review and Tutorial

Optimizing your website for SEO is an ongoing process that requires you constantly monitor it, make changes when necessary, and fix issues as they occur. To do all this, you have to conduct frequent site audits – content audit, SEO audit and technical audits. Screaming Frog SEO Spider is one of the best online tools that helps you perform all these three types of audits and more!

What’s more, it’s absolutely free for websites with up to 500 pages, and paid version that unlocks all the features costs only $99 a year.

Click Here – Free 30-Minute Strategy Session

Be quick! FREE spots are almost gone for this Month

Free Quote

It’s completely OS independent so you can use it on PC, Mac, or Linux based systems.

So what is Screaming Frog?

Basically, Screaming Frog SEO Spider crawls your website just like a search engine does.

It points out the exact SEO issues that are affecting your site. So once you have fixed these issues, search engines will find your site error free.

It doesn’t stop at finding issues. There will be cases where even though a page on your site is error free, there will still be opportunities for improvement.

This tool points out such scenarios too!

Among the various functions that Screaming Frog SEO Spider can perform, the most important ones are:

• Crawl level tracking
• Checking outbound and internal links
• Checking Meta data – meta title, description and keywords – for length and uniqueness
• HTTP status codes such as 404 errors and 301 redirects

How to Use Screaming Frog?

Let’s take a look at the four stages of performing a very basic site audit with this tool. (This article is meant be just an icebreaker and you will discover all the features easily with its intuitive interface.)

1st Stage of Site Audit:

The first stage of the audit is to understand thoroughly where your site stands currently. With this information on hand, you will be able to know the exact amount of work your site needs for it to be error free. A snapshot of your site can also be a good reference point for future use.

2nd Stage of Site Audit:

In this stage, you get the overall picture of the basic SEO elements – meta title, meta description, keywords, H1s, H2s etc – of every page that you want the search engines to index.

Screaming Frog SEO Spider collects all these critical information and presents the data on a spreadsheet.

This way, it becomes easy for you check the following and take corrective measures wherever necessary:

1. Each page has unique meta title and description

2. Titles and descriptions are not too long or too short

3. All pages have proper H1s and H2s

4. All H1s have targeted keywords for the page

5. All H2s have related terms or synonyms of the keyword

It’s widely accepted among the SEO specialists that it’s a good practice to carry out this step first on Google Webmaster Tools before using Screaming Frog.

3rd Stage of Site Audit

In this stage of site audit, you will find all the internal broken links. It’s common especially among the old and larger websites to have many broken internal links.

However, you audit must not skip this step in this is not the case.

Essentially, this tool first finds out all the links that return a 404 error and then finds all the links within the copy that points to such links.

You will also revise all the 301 redirects to ensure they don’t a 404 error.

Whether you have intentionally redirected a page using 301 or just used ‘rel=canonical’ tag, you must be sure that these links are working.

The new page that you are redirecting the users or the canonical link that should get all the SEO values must be clearly defined.

At the end of this process, no URL on your site should return a 404 error.

4th Stage of Site Audit

Another section of Screaming Frog Spider report gives you the overall health of all the outbound links.

In most cases, outbound links return a 404 when the webmaster of the other site gives a no-follow tag to your site.

If this is the case, and if the outbound link is really important and relevant to your site, you might want to contact the webmaster and see if the link can be revived.

If not, it’s best to delete such links.

It’s vital that you carry out both the 3^rd and the 4^th stages of this audit with all the diligence it deserves.

This is primarily due to two reasons – search engines don’t link broken links and can penalize you for their presence and your visitors will not appreciate not finding what they wanted to find.Video Player00:0000:00

Like I mentioned earlier, this article is meant to be just an icebreaker and you will find a host of other tools and features as you continue to use this.

Here are some of the features I like:

• Exporting all the reports and other information to spreadsheets
• Perform thorough content audit and find content issues such as duplicate content
• Tracking robot.txt tag to check for nofollow and noindex issues
• Accurately measure file sizes to optimize site loading speed
• Determines the depth of sub-pages – the number of clicks required to reach a sub-page from the home page
• Fully customized source code search to find any component of the website code with a simple click
• Generating XML site map

Other Tools Similar to Screaming Frog SEO Spider

There are many tools out there that are worth mentioning. Xenu’s Link Sleuth is one of the best I have come across. It’s really fast, very powerful, and FREE.

However, this tool requires advanced coding knowledge and works only on a PC.

If you are curious and want to check out other tools, here’s a list of some of my favorites:

• AnalogX Link Examiner
• SEOmoz Pro App
• A1 Site Analyzer
• WinWebCrawler
• IIS SEO Toolkit
• Integrity(Mac)

The Bottom Line

Site Auditing is a crucial part of your SEO activities and is vital to conduct it frequently if you want stay ahead of your competitors.

Since the way search engines read your site is entirely different from the way a human reads, it’s important to have a software that reads your site just like the search engines.

There are various tools out there that you can use to perform these audits.

However, Screaming Frog SEO Spider is one of the most popular tools among both amateurs and professionals alike.

The main reason for its popularity is its versatility and ease of use. And it comes for free!

So go ahead and perform an audit of your site and check how it is faring.

Do let me know your findings and I’ll be glad to interpret it for you.

The comments section is right below. Good luck!