I love data, I love analysing and I love technical SEO. One of my favourite tools is Screaming Frog Spider, it’s an incredibly useful tool for not only digital marketers and SEO’s but also for anyone who works with websites. It can be used for so many purposes which might not immediately jump out at you but the point of this article is to help you understand how you can utilise it to repair 404 errors (broken/missing page) which should help harness all your internal juice/pagerank.
Before I continue, I should confirm that I have no affiliation with Screaming Frog, my views are completely impartial. Also, if you aren’t sure about HTTP status codes, here is a handy guide. It might also be worthwhile checking out JonJon Yeung‘s Tech Tip Tuesday for this week, in which he talks about the need to implement redirects.
Before using this advice, it is important to start a crawl of your website. Enter the URL in the box at the top of the software and away you go. Let the crawl finish so you have the data ready to be organised.
Finding broken pages
Broken internal pages aren’t great for your website from a user experience point of view as well as from a Googlebot point of view. Finding and repairing broken internal pages should be an important part of any website spring clean.
Start off by clicking the column titled “status code” to sort the pages by – as daft as it sounds – status codes. Any 4XX error means that the page is broken.
Repair 404 errors
You can decide to either re-build the broken/missing page or redirect (more than likely a 301 permanent redirect if you are confident that the page has no reason to return) it to the most relevant page you have on your site. If there is no page on your site which you think is actually suitable for a redirect then you should point it to your homepage, especially if there are backlinks pointing to the page from external websites. To check for any incoming backlinks from external domains, enter the exact page URL into SEOmoz’s Open Site Explorer and it will give you a list of links pointing to the page.
Replacing internal links pointing to the broken page
One thing to note before sitting back and being satisfied with your work is that the page will more than likely have internal links pointing to it. If you click the page on your Screaming Frog crawl report, the bottom half of the page will now show some data. As standard, you should see 4 tabs; URL Info, In Links, Out Links & Image Info. Click the “In Links” tab and you will see a list of all pages on your website which point to this broken page, obviously all of these will need changing. Lastly, ensure that the 4XX page is not included in your sitemap.
