Website crawling is one of the first and most important things to take care of for SEO professionals and website owners. If your website is not crawlable for search engine crawlers, they won’t be able to find the pages and, therefore, won’t be able to index those pages on the SERPs.
This means your pages won’t rank for important keywords, and you won’t get any free organic traffic from search engines.
In this post, we will briefly look at website crawling issues, how to identify if you are experiencing problems, what to do about it, and what Google recommends.
Website crawling and technical issues
It is important to understand that the majority of unintended website crawling issues stem from technical problems in your site.
This may or may not be limited to incorrectly using the robots.txt file and the no-index meta tag. But those are some of the first things you should look at anyway.
See if you have the no-index meta tag on pages that are meant to be indexed on the SERPs. In addition, double-check your robots.txt file and make sure it does not list pages that should be ranking on the SERPs.
How reliable is the site colon search string?
You can see how many pages on your site are being indexed in the Google Search Console. But that number may or may not match the number of pages you get by enforcing indexed pages on Google with the site colon search string.
The site colon search string looks like this: site:yourwebsite.com.
If there is a difference between the two methods, go with the number of indexed pages that you found in Google Search Console.
Here is what Google’s John Mueller has to say about it:
“In general, I would not worry about the difference between a site/colon query shows and what Search Console shows.
In practice, the site/colon query is not meant to be used for diagnostic purposes. So sometimes you do see quite different numbers.
And for some sites, you see much lower numbers. For other sites you see 100 times larger numbers.
And essentially, the numbers we show in the site query result are optimized for speed and to give (kind of) a sense of the website. But they’re not optimized to be comprehensive. So that’s where the Search Console numbers would come in.”
Site auditing tools
If you are still in doubt, you can use website auditing tools to see if there are any crawling issues on your website. There are plenty of free tools that do the job for smaller websites.
Here is the advice that John Mueller gave:
“If you’re talking about a site that is about 500 pages or so and after a period of several months only like 100 are indexed, that seems like something where probably you could do a lot better with minimal extra effort.
So that could be something where what I would first of all check is just the technical side of things to make sure that technically everything is okay, that the website can be crawled.
There are some website crawling tools available. I think they’re even free for very small websites, where you can check your site to see if it’s crawlable or not.
And if it’s well crawlable then the next thing I would consider trying to figure out is what you can do to promote your website a little bit better.
And that could be something like encouraging users to come to visit, maybe by buying ads, maybe by working together with someone else for a while just to (kind of) get the ball rolling.”
Website promotion
Actively promoting your website can help expedite the indexing process. This is especially beneficial if you recently changed pages from being no-indexed to being indexed.
Google crawls websites through the network of links (internal and external) from pages to pages. More activity on certain pages may also send signals to search engines that certain pages should be considered.
That’s why driving paid traffic to pages, creating internal links to and from said pages, and promoting those pages on social media websites can all prove to be useful.
One of the most effective ways to send crawling and indexing signals to Google is to build relevant backlinks from credible sources. John Mueller gave an excellent example for local businesses, and how relevant backlinks can play a part in expediting the indexing process.
He said:
“If you’re an e-commerce site or especially if you’re a small local business site, maybe there are local chambers of commerce that would be interested in linking to your website to give you a little bit of extra information.
So that when our systems look at your website, they say, oh, this is actually a legitimate small business. We should try to index everything.”
Conclusion
Most of your SEO efforts go to waste if your website pages aren’t easily crawlable and indexable. Avoiding website crawling issues should always be one of your top priorities, and conducting regular SEO audits to ensure everything is fine is important.
If you do find crawling and indexing issues, now you know what to do.
For more information and SEO tips, stay tuned.
Luke Harniman
