How to Fix Crawl Errors in Google Search Console

How to Fix Crawl Errors in Google Search Console

In the last few years, Google has made several changes in its Search Console, previously known as Google Webmaster Tools.

The giant search engine has released substantial amounts of data that will be very useful for Search Engine Optimization.

Click Here – Free 30-Minute Strategy Session

Be quick! FREE spots are almost gone for this Month

Free Quote

Given that we have lost significant keyword data with regard to Google Analytics, we have no option other than to depend on Search Console.

The most prominent features that were non-existent in Google Webmaster Tools are Links to Your Website and Search Analytics.

Even though Google’s tools may never fully satisfy us, we must admit that they often release valuable information.

The giant search engine has come up with more help docs as well as support resources to help users of Search Console in finding and fixing errors.

Even though the excitement generated by the new developments may be incomparable to the excitement caused by the creation of 10x content or pleasantly watching as keywords jump in rankings, this part of Search Engine Optimization still plays a vital role.

If you make use of Portent’s Epic Visualization to examine how pieces and bits of internet marketing fit together, you’ll discover that fixing crawl errors found in Google Search Console perfectly fits into the overall infrastructure pie.

Layout of Crawl Errors

General layout of crawl errors within Google Search Console has undergone significant evolution in the last few years. Search Console is broadly classified into two:

1. URL errors
2. Site errors

This form of classification is very helpful since the disparity between errors at page level and site level are distinct.

Issues at the site level can be extremely disastrous since they’re capable of damaging the overall usability of your website.

Conversely, URL errors are less disastrous and less urgent because they only affect individual web pages.

1. Site Errors

Basically, they’re high level errors which can entirely affect the site. Therefore, you ought to pay attention to them.

If you visit Crawl Errors dashboard, you’ll see the errors that have been affecting your website for the last three months.

Ideally, Google should give you a Nice score. It’s rare for the search engine to validate your site. Therefore, if you get a Nice score, just know that you’re doing a great job.

How frequently should you check?

Ideally, you should check for the presence of errors every 24 hours. There will be no major problems most of the time, but you shouldn’t get tired. Just imagine the extent of damage that will occur if you fail to check on a regular basis.

If it’s impossible or difficult to check for site errors on a daily basis, consider examining your website every 90 days to find out if any problems may have occurred during the 3-month period, and fix the errors accordingly.

Site errors can be broadly classified into three:

1. DNS errors
2. Server Errors
3. Robots failure

2. DNS Errors

What are DNS errors?

DNS is an acronym for Domain Name System. They are the most common errors and are usually the first to affect websites.

If Googlebot experiences issues relating to DNS, your site will be unable to connect with Googlebot through a DNS lookup or timeout issue.

What is their importance?

DNS issues play a very important role since it’s the first step that must be taken when you need to access your site.

You ought to take drastic and firm action whenever you start experiencing DNS issues which prevent connection between your site and the giant search engine.

How can you fix DNS issues?

Google recommends use of a tool known as Fetch as Google to find out how Googlebot crawls on your web pages.

If you only need a DNS connection status, you can Fetch as Google without necessarily rendering, especially if you want quick results.

Check with your service provider of Domain Name System.

If the search engine is incapable of properly fetching and rendering your page, then you’ll need to take more action. Ask your DNS service provider to help find out where the problem is.

Make sure the error code displayed on your server is either 500 or 404 error. Your server ought to show a Server Error or Not Found Error (500 and 404 respectively).

With regard to accuracy, the two aforementioned codes are better than experiencing a DNS error.

Other tools that you can use to check for and fix DNS issues are Web Sniffer. net and ISUP.me as well.

3. Server Errors

What is a Server error?

A server error is that which occurs when the response time of your server is extremely long, leading to a timeout of the request.

Are you aware that the Googlebot that crawls your website can only wait for a limited period of time when there’s need to load your site.

If your server’s response time is very long, Googlebot will simply give up.

There is a difference between DNS errors and server errors. In the first case, Googlebot experiences difficulty in finding your URL, courtesy of DNS issues.

In the second case, Googlebot server issues prevent Googlebot from loading the web page, even though it can connect to the website.

What is the importance of Server errors?

You must take action immediately as you discover that your site is experiencing server errors. This is because they can have disastrous effects on your site.

The errors make it impossible for Googlebot to crawl. As a result, it consistently gives up after a certain period of time.

How to fix server errors

Google recommends use of Fetch as Google to find out if Googlebot is capable of crawling your site.

If the tool indicates no problems with your homepage, it will be safe for you to assume that the search engine is capable of accessing your website.

There are various types of server error issues. They include; timeout, no response, truncated headers, connect timeout, connection reset, connect failed, truncated response and connection refused, among other issues.

Find out the specific type of issue that affects your website before attempting to resolve it.

Robots failure

This error occurs when Google fails to retrieve your site’s robots.txt file.

You’ll be surprised to know that the file is only required when you want the search engine to skip particular pages when crawling your site.

You only need the file if your website comprises of web content that ought not to be indexed by Google or other search engines.

What is the importance of Robot’s failure?

Urgent action need not be taken if your site is still small and very static. Additionally, you need not worry if you haven’t added any new pages or if no significant changes have been made in the recent past.

Conversely, if your website publishes new content nearly every day, the issue must be fixed within the shortest time possible.

This is because if Goglebot fails to load the aforementioned file, it won’t crawl your site. As a result, any changes made or pages added won’t be indexed.

How to fix robots failure?

To fix the problem, you must first ascertain proper configuration of the robots.txt file.

Confirm the pages that you want Googlebot to skip while crawling, since it will crawl all the other pages by default.

Additionally, you must check for existence of the Disallow clause. Make sure it doesn’t exist at all.

If your file exhibits no problems but you still experience problems, use a tool known as server-header checker to find out if the file is displaying a 404 or 200 error.

4. URL Errors

There’s a big difference between site errors and URL errors. Whereas the former can affect the whole website, the latter can only affect certain pages.

Many owners of websites have experienced issues brought about by URL errors, and the issues sometimes make them worried.

However, if you own a site or sites, you don’t need to freak out. Just remember that Google usually ranks errors depending on the damage they can cause.

Additionally, the errors may automatically get resolved

If you have made substantial changes to your website in the recent past with the objective of fixing errors, or you strongly believe that most of the errors stopped occurring, why not mark all the resolved errors, and keep tabs on them on a regular basis?

The benefit of taking this action is, the errors will stop appearing on the dashboard, although the search engine will keep revealing them whenever Googlebot crawls your site.

If the URL errors were properly fixed, they will not appear again. In case they appear, you must fix them within the shortest possible time.

Soft 404

This error occurs when a web page is indicated as found, instead of not found.

Are you aware that a 404 page may look like a genuine not found page, even when that isn’t the case?

The user-visible element of a typical 404 page is the page’s content. The displayed message should inform users about disappearance of the page.

Website owners usually help site visitors by availing interesting 404 responses or list of web links that they can visit.

What is their importance?

In case the pages listed as 404 errors are not very important, there’s no need for you to urgently fix them.

However, if the pages listed as 404 errors are very important because they contain product categories, address of your business or information concerning payment for goods or services, then it will be necessary to fix the problem as soon as you can.

How to fix Soft 404 errors

If the pages do not exist any more:

1. Make sure the server-header-response is either 410 or 404. It shouldn’t be 200.
2. Redirect all the old web pages to relevant pages within the site.
3. Avoid redirecting large volumes of dead pages to the website’s homepage. Instead, direct them to similar pages that you’ll consider appropriate.
4. 404 Errors

This type of error indicates that Googlebot attempted to crawl on a non-existent page. It indicates the error when the non-existent page is linked to existing web pages or sites.

Google clearly indicates that 404 errors have no effect on rankings. For that reason, you can ignore the errors.

What importance do they have?

If pages that contain very useful information display 404 errors, immediate action must be taken. However, if the web page disappeared but it contained no useful information, then you don’t need to worry.

Fixing 404 errors

The following are steps that you should take if your web pages display 404 errors:

1. Make sure publishing of the web page is done from your CMS (content management system). It shouldn’t be deleted or published in draft mode.
2. Make sure the error URL isn’t a variation, but a correct page.
3. Find out if the error appears on your website’s http versus non-http and www versus non-www versions.
4. Ensure you 301- redirect the affected page to a related page that you consider most appropriate.
5. Access denied errors

This message is an indication that Googlebot is unable to crawl the web page. The errors prevent it from crawling in the following ways:

1. When you ask users to first log in before seeing a URL or,
2. Robots.txt file prevents Googlebot from crawling specific sites or,
3. Your host blocks Googlebot, or severs asks users to confirm by proxy.

Role of Access Denied errors

If you’d like Googlebot to crawl the blocked pages, then you’ll need to fix the errors within the shortest time.

However, if you don’t want it to crawl the blocked pages, just ignore the error messages.

Fixing access denied errors

1. Get rid of the login requirement from pages that you’d like Googlebot to crawl on.
2. Find out if pages listed on robots.txt file are supposed to be blocked from both indexing and crawling.
3. Use a tool known as robots.txt tester to find warnings and conduct tests on individual URLs.
4. Use Fetch as Googleto test your site’s appearance to Googlebot.
5. Make use of Screaming Frog to scan your website.
6. Not Followed Errors

These errors occur when Google encounters problems with Flash, redirects and Javascript, among others.

You should be worried about these errors when they occur on high-priority URLs.

If the problems arise from URLs that have become inactive, or stem from non-indexed parameters, then there’s no reason for you to worry.

To fix the errors, make use of Fetch as Google or Lyn Text Browser to examine the site. If you fail to see important content or the pages load, you would have discovered where the problem lies.

For not followed problems concerning redirects, follow the steps outlined below:

1. Find out if there are redirect chains.
2. Update the website’s architecture.
3. Get rid of redirected URLs in the site-map.

There are other tools that you can also use to fix these errors. They are Screaming Frog Search Engine Optimization Spider, Raven-Tools Site-Auditor and Moz Professional Site Crawl.

Server Errors and Domain Name System Errors

Google still classifies DNS and server errors under the URL errors umbrella. This is primarily because the giant search engine wants you to detect and fix the errors the same way you’d take care of the errors mentioned above.

For this reason, please refer to the information contained above.

Final verdict

Some issues concerning Search Engine Optimization can be extremely boring, especially if they’re technical in nature.

It can be quite difficult for someone to start inspecting URL errors that may seem inconsequential.

On the other hand, nobody wants to freak out as a result of discovering numerous errors on their website.

However, if you acquire this habit, you’ll have the capacity of reacting to these errors in the right way. As a matter of fact, the experience will make you capable of handling the errors with minimal mental effort.

Google has availed lots of information concerning Search Console. Access the information and take your time to go through it.

The formidable search engine offers extremely effective and efficient tools for identifying and fixing errors. If you fix the errors, there will be great improvement in your site’s rankings.

Your site may even appear in the first page of Google’s search results.

Additionally, fixing the errors will enable people who visit your site have better browsing experience and also enhance conversion rates. What are you waiting for?