How to crawl and index blogger website on google

How to crawl Blogspot domain on google | index your website posts on google | rank your blogger website on google search engine 

Hello guys,  today in this post I will teach you how you can easily index your newly made blogger website on google to rank its pages and get decent traffic.

What is need to index the website on google ?

Well, the answer is simple, without indexing your website on any search engine platform you can't get traffic, you cant serve your content to the people out there on the whole internet, and lastly, you can not generate any revenue from the content you have published on your website. That's why it is necessary to submit your website on google so that it can be indexed by it and you can reach out your content to all the people out there who are in need of it.

Why choose Google only?

So now you all would be thinking why to chose google search engine to submit our sites for indexing, there are other platforms too, like yahoo, bing, etc...

The answer is as simple as that, Google is so dominant that the term "Google" can also be used as a verb, so that when someone searches for something on Google, they may say they "Googled" it.

Google is an internet search engine. It uses a proprietary algorithm that's designed to retrieve and order search results to provide the most relevant and dependable sources of data possible.

How to crawl  the blogger website  to rank in google,  and get post indexed on the google pages

Applying for crawling of the website is not a   difficult task, but if it is done in the wrong way, may result in problems in the future with the indexing of your newly published post on your blog. 

So I will appeal to you to go through the right steps that I will mention and read every step carefully.

Applying your website for indexing in google will go through the following steps: - 

1. Submitting the sitemap 

2. Creating Robots.txt 

3. Making some changes in blogger settings.

So we will start the guide from our very first step 

Submitting the sitemap  of your website on google search console 

Go to google and type  this website https://www.labnol.org/blogger/sitemap  and copy-paste the URL of your website,  click on generate the sitemap and copy all the things after  / (slash mark)  which starts from atom

example :-  it will look like this   atom.xml?redirect=false&start-index=1&max-results=500 

Now again go to google and type google search console open this website sign in there with your Google account, next a window will open, here you will have to enter the web address of your website. Once you successfully do all these steps, you will be prompted to the dashboard of your google search console.

Now scroll down, and look for an option named sitemap,   in this section you have to enter the sitemap of your website which you generated above with the help of the sitemap generator. 

this was  our sitemap  👉 atom.xml?redirect=false&start-index=1&max-results=500

You have to enter this sitemap in the sitemaps option in the google search console and then press submit. 

Now check out the status option, if it shows success then congratulations, you have successfully submitted the sitemap of your website. And if it doesn't show success then you would have done any mistake during these steps, so repeat the steps again from the starting. 

Creating Robot.txt 

Next, you have to create a robot.txt file and then you have to submit it in your blogger settings.

So go to this website https://smallseotools.com/robots-txt-generator/   don't touch any other option here, just search for an option named sitemap, and here you have to fill your website's web address along with the same sitemap you generated for google search console. 

It will look something like this 👉 https://www.mysite.com/ atom.xml?redirect=false&start-index=1&max-results=500

When you finish doing all these steps, click on create Robot.txt, next you will find your robot.txt file generated on your screen.

# robots.txt generated by smallseotools.com

User-agent: *

Disallow: 

Disallow: /cgi-bin/

Sitemap:  https://www.mysite.com/ atom.xml?redirect=false&start-index=1&max-results=500

Your robot.txt file will look something like this, but instead of mysite.com, it will show the blog address that you would have written above. 

Now, copy your  robot.txt   file and open your blogger. In the blogger dashboard, go to settings options search for crawlers and indexing section. Next, click on Enable custom robots.txt. Here you have to paste your robot.txt file once you do it successfully click on save.

Making changes in blogger settings. 

This is the very last step of crawling and indexing. You have to go to the dashboard of your blogger again and then click on settings options. Here, under the crawlers and indexing section, you have to click on enable Enable custom robots header tags. 

Next, click on Home page tags and enable 'All',  'noodp' 

Likewise, click on Archive and search page tags  here again you have to select  'All',  'noodp'

At last click on Post and page tags  and this time you have to enable  'All',  'noodp'

So 'All' and 'noodp' are common in all three hence make sure to enable them correctly.