Not indexed? How to index an Online Store in Google

How to Index a New Online Store?

The worst position a client can be at is having an online store which can't be found on Google for any keyword, not even their brand name. The solution to this situation can be very easy, especially if the online store is brand new. A new online store can be unknown to Google and submitting a sitemap into the Google Search Console can solve the problem. The complete online store could also have set their meta robots to noindex or it has forbidden crawling for the Googlebot through robots.txt, but it also can be a more complicated situation where that particular domain was used before and was known for using illegal or shady SEO tactics better known as gray or black hat SEO. Sometimes the only solution to such obstacles is changing the domain. To be able to always solve these kinds of situations it is important to understand the SEO basics and how search engine crawlers like the Googlebot work.

Back to the Basics

Back to the basics. Too many people just jump into SEO and never learned the basics thoroughly. Very often their best case scenario is that they have heard about all the basics collecting SEO knowledge partially through a certain period of time and experimenting on projects. That isn't a bad approach at all but sadly most people lose too much time chasing some SEO tricks and new tactics from Reddit threats and shady discussion boards hoping to find the hidden secret of successful SEO enthusiasts, which at the end won't give them instant success. Well, the biggest trick or secret most successful SEOs know and don't like to tell you is that you need to invest a lot of time into understanding all ranking factors and getting an overview to understand how they are all connected and that you need a lot of experience to be able to understand which of them is more important than others, depending on the situation you find yourself in. They don't tell you that because nobody wants to hear that it will take much longer than they imagined before they get to a point where you have a founded self-confidence regarding SEO. It is way easier to sell you some clickbait articles and videos with the newest trick, but if you have already tried out some of those advices, you have may have noticed that most of them almost never work, but many still keep collecting pieces of knowledge from those sources, because you are chasing the one trick that will make you successful in SEO. For all of of you who want to have a deeper understanding of this topic I will try to give you an overview of many possibilities, so you can solve such situations without playing a guessing game. So in this lesson you will learn how the Googlebot works based on what I call them: the magic SEO circles. The Google bot knocks, he obeys certain recommendations, but some of them he might ignore. He crawls, calls for help, has a crawling budget and can get easily lost in translation, but still he is predictable and controllable.

What Happens at the First Googlebot Visit?

Let's start with understanding what happens if the Googlebot is visiting a new online store. If a new online store gets published on a fresh registered domain, Google can't know that the online store even exists until one of three things happen. First, you submit your online store in the Google Search Console and request indexing. Second, the online store gets a backlink on a frequently visited website which is also being frequently visited by the Googlebot who will find the backlink and follow it. After the Googlebot visits the new online store and is able to crawl it and doesn't find any indexing restrictions, he will index it. And third, you post the URL of your new online store to social media channels like X who have a direct connection to Google which will trigger to the Googlebot too. To be able to understand what are considered obstacles for the Googlebot, both for crawling and indexing an online store, you need to understand what the domain name, DNS servers, IP addresses and HTTPS requests are and how they are all connected.

Understanding How the Googlebot Works

A detective is at hotel reception, he got the suspect's name through a call from a reliable informant, or he saw the suspect on a surveillance recording or the suspect called the police by himself, or the detective saw a social media post mentioning the suspect. He knows only the name of the suspect and nothing else. He needs his room number and to search his room to identify who the suspect really is, so he asks for the suspect's room number on the reception and the nice lady tells him that the suspect is in his room and gives away his room number, and if he occupies the room by himself or with others. If the room number is 404, the detective already knows that no one is there and returns to the station. If there is no receptionist, he will write the room number 500 down and come back again later. If everything is okay, the detective takes the elevator to the suspect's floor and finds the door. There is no sign on the door, so he knocks and asks for the suspect. If the suspect is under the shower, he will tell the detective to come in and he will be ready soon. If not, the suspect will open the door and the detective will judge on his first impression if it is safe to enter his room and interrogate him. If he decides to enter, he will ask him first about his profession. If he tells the detective that he is an online store owner, he will ask him what he is selling and try to narrow down an overall product range for his online store. Then he will try to find out if the suspect knows a lot about those products and how many products he has in each category and he will check if he is using Google analytics to track his sales in order to check later how big the revenue is. If everything is reliable, the detective will strike him from the suspect list and allow him to be listed inside the national phonebook to be found under his domain name.

Common Indexing Obstacles for the Googlebot

What could have gone wrong during the interrogation which is the crawling process and prevent the detective Mr googlebot to allow the suspect's company name, which is the online store, to be in the phone book, which is Google? First of all, the online store could be unreachable. Secondly, there could be multiple domains with the same IP address, so the Googlebot would need to invest more time to open the online store and if the other domains would be suspicious, like porn not gambling websites, in that case the Googlebot would be more careful. There could be a sign on the door which disallows the Googlebot to enter the online store. There could be also on "noindex", "nofollow" sign inside the online store, which would discourage the Googlebot from further crawling. And if the Google bot is welcome to craw the online store, he will first try to find out what the online store is selling, who is the seller and how much products in the different categories the online store has and how much of which products that online store actually is selling. If it is very clear what the online store is selling and that they have enough knowledge about their products the Googlebot will be happy to index most of the online store. If it isn't clear what the online store is selling exactly, the Googlebot will index at least the homepage and come back again later to see if the entity became clearer. In the next lesson we will take a deeper look what to do if the Googlebot decides not to index an online store after crawling it, although the online store has a clear entity and most of the basics things sorted out. Now we know that most new online stores are not indexed because no one called the Googlebot to take a look. We can change it by tweeting about it, submitting it into the Google Search Console or getting a backlink from a well known website, like from a news article or a simple blog article or getting in any way many visitors to the online store, which will be seen through using the Google Chrome browser .If you have got value out of this lesson, please consider subscribing to our channel.

How to Check If Your Online Store is Indexed?

If the Google Search Console is not ready or you don't use it at all, you can always check if your domain is indexed in Google or not by using the site parameter. In this way you will see a part of all index URLs of your online store. That is almost never complete and should not be used to control how many URLs are actually indexed. For that purpose you need to use the Google Search Console. In the next lesson we will see how to fix the particular problem when Google is crawling an online store, but not indexing it. To understand why that is important let let us take a look at the already mentioned magic SEO circles. But before we begin you should know that you can control crawling and indexing process with meta robots and canonical tags. We will cover that topic later in the SEOLAXY SCHOOL and learn how to manage the crawling and indexing budgets in SEOLAXY ACADEMY later. So let's take a look at the magic SEO circles.

The Magic SEO Circles

So we we are now seeing this one circle is representing all the URLs of your online store. They are either crawlable or not crawlable. That means you can forbid the Googlebot to enter parts of your website and allow it for the other parts of the website. So allowing and disallowing through the robots.txt file. After that we know a part of the crawlable URLs and a part of the not crawlable URLs are set to index and some other a smaller hopefully part is set to not indexable. In the next part we know which URLs Google actually crawled and they most of the times are very strictly following the guidelines of the robots.txt, which means they're crawling only the crawlable things which you didn't disallow through the robots.txt, but they're crawling also the not indexable URLs, which you have put on "noindex", but they're still crawling it. So putting an URL on "noindex" doesn't mean that that URL won't be crawled. So the next thing we need to know that just a part of those URLs which are put on "index" and have been crawled by Google will actually get indexed. So not of all URLs which you put on index will be indexed, especially because a part of them is not crawlable and the other part is not good enough. But about that we will make a lesson later on and about controlling the meta robots text "index" and "follow", and about the canonicals and so on and so on. Now we're here to understand where we are getting at. So we're getting an overview of the situation which is possible where we want to come to and which circle, which magic SEO circle is the most important of all. So the next one is out of those URLs which are indexed in Google. Only a part of them will rank actually in the top 100, in the top 20, in the top 10. So that means, like you see, a very small portion of your URLs will rank in Google and just a few URLs, which actually rank, will get impressions, will be seen. So if you're not ranking inside the top 10, the possibility that you will get impressions from that ranking are very low. So ranking on page number one or among the first 10 results is getting you impressions but that doesn't mean that all of those URLs which are ranking and which are seen by users are getting clicked. So just a part of them gets visits. So, you see, the circle is getting smaller, and smaller, and smaller. And out of those URLs which are ranked, are seen, and are getting visits, just a few of them will bring you new customers. So the rest of the URLs will get you just visitors. They will go through your website and not buy, but just for a few rankings you will get also visitors which will convert to buyers.

Summary

So now you've got an overview with the magic SEO circles that everything is connected, that a Googlebot visiting your online store, crawling it, doesn't mean he will index it and if he indexes it, it doesn't mean that you will rank and if you rank it doesn't mean that you will have visits, and if you have visits it doesn't mean that you will have buyers. That's very interesting and very important to distinguish, to know, that there is a big way to go to discover the keywords which are actually bringing you some revenue.

Next Episode Preview

So in the next lesson we will go one step further and investigate why Google sometimes visits an URL but decides not to index it. It could be many reasons and we will take a look at the most important reasons and how you can remove those obstacles and get the desired effect inside Google. So if you got value out of this lesson, please consider subscribing to our youtube channel, and maybe liking and sharing it with others. See you next time.