Preventing Search engines indexing pages on your site

They may be situations when you may not want Search engines such as Google and Bing to index pages on your website. For example login pages, testing sites or basket pages. I’m going to list how to do this using the Robot.txt file and add a meta tag to the page.

Robots.txt

Disallow: /login
Disallow: /register

The robots.txt is an optional text file which lives in the root of your website directory. The Robots.txt file is used to suggest search engine crawlers which page to access on your site.

Using the robots.txt method for the pages you don’t want to show up in the search engine. In the txt file add a new line which says Disallow followed by the URL that shouldn’t be indexed. So it looks like the example above.

One thing worth mentioning is documented in Google Search Central. You shouldn’t rely on just this method as if a page has been linked to another page. Google may still index the URL even though it has visited the page. Google suggest also using the following method we are looking at which is noindex tag.

Meta tags

<META NAME="robots" CONTENT="noindex,nofollow">

On the pages, you don’t want to index if you add the following code on the page. Into the head tag of your website. The code tells search engine robots not to index the page, which is done by noindex. The nofollow tells the search not to follow any of the links on the page. Depending on your circumstance you may be able to omit this.

Further reading

• https://developers.google.com/search/docs/advanced/robots/intro