Everyone wants to be on Google’s first page, maybe even the first place? There probably isn’t a website owner out there who doesn’t dream of it.
After all, you invested a lot of time and energy to create the best possible website. Why wouldn’t you want your effort to be rewarded?
However, you know what’s the first step to getting to Google’s pole position? Getting the site indexed at all. Sometimes websites don’t even make it that far.
In the online universe, that’s pretty much a disaster. If you are not picked up by search engines, your website is going to be invisible for 99.999% of the people on the Internet and you can completely forget about organic traffic.
Granted, there are alternate traffic sources, but visitors from search engines remain one of the most important and potentially profitable options.
In order to avoid this kind of catastrophe, in this article, we will look at reasons why you may be shunned by Google and then over a step-by-step approach to ensure this doesn’t happen.
How Does Google Find My Site?
Before we get started, we need to settle on some search engine basics and important vocabulary. This will make it easier for beginner to understand the following.
To find content for their search results, Google relies on so-called search spiders. These are automated programs that scour the Internet and report the content of websites to search engines.
All major search engines have their own spiders (for example, Google’s is called Googlebot, and there are many more out there.
The process of exploring a website and its content is called crawling. It’s what the Googlebot does to understand what your posts and pages are all about.
The phase after that is called indexing. It means if your content has been deemed worthy by the snarky little robot, it is then processed and added to the Google index. This index, in turn, feeds into the search results
What If My Site Isn’t Indexed By Google?
The first step to see whether your site has popped up on Google’s radar is typing site:yourdomain.com into the Google search bar (naturally, exchange yourdomain.com with your actual domain).
This will show you all the pages under your domain that Google has on their index. If the results are completely empty, that means the search engine is either completely oblivious of your content or something is going on with your site that keeps it from being indexed.
This can happen for several reasons:
- There are problems with your server such as technical issues or faulty configuration
- Your website violates some of Google’s terms and has been penalized
- Your WordPress site or its SEO settings are configured badly
- You haven’t taken any measures to get indexed or even noticed
- The site is too new and Google hasn’t had enough time to add it to their index
Every case is different and there can be more than one issue at play. However, don’t worry, though we will go over each of these factors in detail below.
Also, note if you are here because you would like to improve the rankings of your already-indexed posts and pages. That is a different case and you might want to check out these articles .
Making Sure Your Website Gets Indexed by Google
In a way, we are completely at Google’s mercy when it comes to being found on the web. No index, no organic traffic.
However, the good news is that there are plenty of things we can do to move Google into giving us a spot in their SERPs which we will talk about in the following.
Use A Good Hosting Provider
One of the first potential barriers to getting indexed by Google is the hardware your site runs on.
Slow server speed, downtime, and disconnects can cause search spiders to abandon their cause. While not very common, it is a possibility.
Since in hosting you get what you pay for, investing in a quality host with good hardware and excellent availability is always worth it.
Write High-Quality Content
The thing about being indexed by Google is, we don’t just want them to be aware of our site, but be aware of it in a good way.
If your site is empty or — worse — full of crappy content, it might get indexed but it won’t get anywhere near the front row of the SERPs. That’s almost as bad as not being indexed at all.
It’s no secret that Google cares about the relevancy and quality of your content. For that reason, when you set up your site, focus on high-quality, useful, original content. Naturally, that also means to stay away from duplicate and/or scraped content.
Check Settings In WordPress
During the development phase, usually the last thing we want is to be indexed by search engines. In fact, we want to keep Google and Co as far away from our site as possible.
Otherwise, we may be caught with incomplete (and thus low-quality) content and Google will form an opinion about our site based on that. No good.
The problem is only when we forget to revert the measures we have taken to keep search engines away after our site goes live.
One of the most common mistakes is to leave “discourage search engines from indexing this site” active in the back end of WordPress. That’s basically a death sentence for organic traffic on your site.
So, in order to make sure you get indexed (or if you are having problems appearing on Google), definitely have a look at this setting at the bottom of Settings > Reading to make sure it is unchecked.
Don’t forget to save if you made any changes.
Set Up The robots.txt file Correctly
Telling WordPress to discourage search engines does a bunch of things. For one, it adds noindex and nofollow meta tags to the head of your website (more on that below) and also keeps anyone from pinging your site.
Most importantly though, this setting configures the WordPress-internal robots.txt file to disallow all search spiders from indexing your site.
In case you didn’t know, robots.txt is an important server tool to directly communicate with search engines. You can use it to tell them which parts of your site you want them to index and which you don’t.
Because of that, you or your developer will often set up an additional robots.txt file to control indexing during development. While that’s fine and dandy, it becomes a problem when you forget to change it back.
Share Your Site Online
As I said earlier, in order to index your site, search engines first need to find it.
How do they do that? In short, links.
When an already-indexed page points to your site, it becomes much more likely that Google will find you as well.
So, what is an easy way to create a link to your site?
The answer: social networks.
By sharing your website on Facebook and Twitter, you can alert search engines (and fellow humans) that there is a new site in town.
While it’s true that many of these links will usually be set to nofollow, search engines that track social signals (like Google) should still be alerted to your presence.
Sign Up To Google Analytics
Regardless of indexing, in order to gather more info on the performance of your site, you will want to install some sort of Analytics
In that regard, most of us opt for Google Analytics because it’s free, comprehensive and gives us all the necessary information to work with Google.
However, there is another good reason: When you set up Google’s analytics suite, you also tell Google that there is a website they should pay attention to which makes it more likely that they will.
And if you are feeling adventurous and want more advanced tuning, you might want to have a look at SEMRush, its got an absolute ton of tools in there which can be used, but that’s beyond the scope of this article.
Create A Sitemap
A sitemap is an XML document that contains a list of all the content on your site. It tells search engines what you have to offer and how often they should check back to see if there’s anything new.
In short, sitemaps are a great tool for guiding search spiders and creating one for your website is a good idea.
To do so, we have several plugins at our disposal. One of the most popular is Google XML Sitemaps which is trusted by more than a million users.
So just go for one you feel comfortable with and tweak it until you get it right.
Submit Your Site To Google
However, the usefulness of sitemaps doesn’t stop there. After all, do you just want to let it sit around and wait for Google to discover it?
Here’s a better idea: How about letting the search giant know exactly where your sitemap is located so they can start indexing your content right away?
You can do exactly that in the Google Webmaster Tools. If you don’t have an account yet, use the link above to set one up.
Google Webmaster Tools give you a lot of information about what Google knows and thinks about your site and how you can further improve it.
After connecting your site to the Webmaster Tools, go to your account and access Crawl > Sitemaps. Here, click on Add/Test Sitemap in the upper right corner.
Now all you need to do is add the address of your sitemap to the end of your domain. It’s usually something like yourdomain.com/sitemap.xml.
Set Your Preferred Domain
While we are in the Webmaster Tools, why not use the opportunity to also set up your preferred domain?
Why so? You see, your site usually exists in two different domain forms, namely http://yourdomain.com/ and http://www.yourdomain.com/.
While both point to the same location and both are fine for Google, technically it is not the same domain. Instead, the www version is a subdomain.
If you don’t submit both sites to Google and tell them which one you prefer, sometimes you will get a message that your website isn’t indexed even though it is but the wrong version.
To avoid this, make sure you add both site versions to your GWT account (from the Search Console dashboard). After that, access both web properties and use the settings button in the upper right corner to go to Site settings.
Set both sites to the same preferred URL (your choice) and Google will from then on stick to one version for crawling and indexing.
Now all you have to do is set a redirect on your site that points people accessing the non-preferred version to the preferred one. Done.
Check For Crawl Errors
When indexing fails, it can often be because there is something technical wrong with your site. Thankfully, if that is the case, Google Webmaster Tools will warn you about it.
It will do so directly on your dashboard (the big Crawl Errors window) and under Crawl > Crawl Errors.
Most often errors will be 404s, meaning links to URLs that don’t exist.
It’s fine if there are a few of them (it happens), however, in this place you will also notice if there is something bigger going on that keeps your site from being indexed.
This information is crucial for taking remedial action and you can find similar notices in your Sitemaps menu.
Request Google To Crawl Your Site
If you do have problems on your page and have taken steps to correct them, you can then ask Google to kindly have a look at the affected pages again.
This way you will know whether your solution worked and also tell Google that these pages are ok now without having to wait for them to crawl your site in their own sweet time.
To do so, you first need to fetch the URLs in question. This happens under Crawl > Fetch as Google.
Here you can input any page on your domain for Google to check and via Fetch and Rendereven display it the way their search spiders see it. The latter can help you spot errors or other things that go wrong during processing your page.
Once you have fetched a page, it will appear in the list at the bottom with a button that says Submit to index. If the fetching and rendering goes well, you can use this to tell Google to try and add the page to their index.
You can either submit only the page itself or include all direct links it points to. You are allowed to add 500 pages per month with the first option and 10 with the second.
Check .htaccess
.htaccess is another important file on your server with info for browsers and search engines how to use and deal with your site.
For example, .htaccess contains rewrite information, which is enables you to use pretty permalinks. Besides that, it can also contain permissions for different directories on your site.
While that’s super useful, it also means that this file can be a problem source for your site’s indexing efforts.
It’s a bit beyond this article to go into this (very technical) topic, however, if you are experiencing problems with indexing that you can’t explain otherwise, checking this file might be worth it.
Sometimes it can be enough to go to Settings > Permalinks in the WordPress back end and click on Save Changes to have WordPress flush the .htaccess file. If this step doesn’t work, you may have to do some googling.
Check Meta Tags
Meta tags are elements in the code of your site that provide additional information about its content and architecture.
For example, if you set a meta description in a WordPress SEO plugin, it will show up inside a meta tag in the head section of your page.
One tag named robots in particular is very important.
Its value can be index/noindex and follow/nofollow. As you can probably guess these are directives aimed at search engine spiders.
In fact, the aforementioned “discourage search engines” option in WordPress adds a noindex, nofollow tag to your site until switched off.
If you are experiencing indexing problems (or just want to make sure there isn’t a problem), you can check the head of your site via Firebug or your browser’s developer tools.
Should meta tags like that exist, you will find them quite easily. After that, it’s only a matter of figuring out which plugin or program is responsible for putting them there.
Get Quality Backlinks To Your Site
As mentioned earlier, search spiders usually find a site through a direct link.
However, links are not only a pathway to your website but also a way for Google to judge its quality.
While in earlier times you could go to any kind of web directory and shoot yourself links until you ranked high, today this kind of behavior will hurt you more than help you.
By now, it’s all about link quality.
Google determines a high-quality link by the following characteristics:
- Relevancy — The link is coming from a site related to your topic or industry
- Trustworthiness — Not from a low-quality or spammy website
- Activity — The link actually sends traffic your way
- Relevant anchor text — The text that functions as a link (like this one) is meaningful
- Link location — Links inside an editorial piece carry more weight than sidebar or footer links
- PageRank — If Google already trusts the linking page, they will also feel good about websites it links to
- Uniqueness — Different websites that refer to you once are more valuable than one website linking to you over and over again
- Reciprocity — If the link is a one-way street (meaning you don’t link back to the same site), it means there is no link exchange scheme going on
That’s a tall order, I know. But be aware that not every backlink has to have all those characteristics to help you in your ranking.
Acquiring high-quality backlinks is an art form in itself, and Backlinko has written an excellent guide on how to do it.
It’s also more of a long-term strategy and not a quick fix, but it is worth it. Not only will it help you get found but also make your site more valuable in Google’s eyes.
And that’s a good thing because the more valuable Google thinks you are the more pages they will index from your site.
It Can Take A While..
Patience doesn’t come easy to everyone, but sometimes there is no other way.
Even if you do everything right, it’s unlikely that Google will index you two minutes after going live. That’s just not how it works.
While it is possible to appear in search results within 24 hours, there is no guarantee that this will actually happen.
So, if you are the impatient kind and have done everything you can, go read a book, take a walk, eat some doritos or do something else for a while until it’s time to check the results of your efforts.
How To Get Indexed In A Nutshell
One of the first steps towards search engine success and free organic traffic is getting indexed by Google. Without being picked up by search engines, all the SEO in the world will not do you any good.
Fortunately, there are a lot of things you can do to make getting indexed more likely. From correct server and WordPress settings to content optimization and backlinking — the possibilities to improve your chances are numerous.
Following the above tips should lay a solid groundwork for appearing in the SERPs. You can also use many of them for troubleshooting if you are experiencing problems.If you want to ramp things up a bit, and are prepared to get your hands dirty with some analytics and tweaking, then I’d recommend SEMRush, go and get the free trial of that to start with.
If you enjoyed this post, why not check out this article on the Best new SEO software!
Post by Xhostcom Wordpress & Digital Services, subscribe to newsletter for more!
