Google To Officially Stop Supporting NoIndex Directives from Robots.txt


robots.txt
Listen to this article

The search engine giant is officially going to stop obeying robots.txt noindex directive starting 1 Sep 2019. Publishers still using robots.txt noindex directive will be required to use an alternative to stop Google from crawling or indexing their pages. According to the company, Google never documented unsupported robots.txt rules such as no follow, crawl-delay, and no index. And since Google Bots used to support it unofficially in the past, it will not be the case hereafter.

Google announced to withdraw robots.txt directive support on 2 July 2019, through a post on Google Webmasters Blog. The company confirmed that robtos.txt noindex was never officially supported by Google and starting today, the crawlers will also stop supporting it. Google shared the blog through twitter bidding its goodbyes to the unsupported robots.txt rules.

google web masters on twitter

Alternate Ways From Google to Control Indexing

Google published 5 alternative methods to control crawling on its official blog:

  • Noindex in robots meta tags: While robots.txt no index is no more supported, meta tags with no index directive in either HTML header codes or HTTP response headers have become the most effective way to stop URLs from indexing. 
  • 404 and 410 HTTP status codes: These status codes inform the search engines that the page on respective URL does not exist. Google automatically drops these URLs after crawling them once.
  • Password protection: If a page is hiding behind a login, Google will drop it from the index unless the password protection indicates either paywalled content or subscription.
  • Disallow in robots.txt: If the search engines don’t know about a page because they were blocked from being crawled, it implies that their content won’t be indexed. Search engines may index a URL if other pages have links to it but Google aims to make pages less visible if the crawlers cannot see the content.
  • Search Console Remove URL tool: The tool is widely known for temporarily removing a URL from Google’s search results.


Make sure you are not using robots.txt no index on any pages. If you are, we recommend you to immediately use one of the above-mentioned methods to avoid your pages from indexing.
 

Related Post

digital marketing strategies
blog
Top 5 Digital Marketing Strategies for Dentists

Listen to this article Are you a dentist who recently opened a new dental clinic and is searching for quality leads? Are you facing slow traffic and feeling clueless about how to attract more patients to your dental clinic? As a dentist, establishing a strong

...
Google AdWords
blog
Know The Benefits of Advertising With Google Ads To Grow Your Business

Listen to this article The Google Ads publicizing program has progressed significantly as of late, giving sponsors numerous more apparatuses and alternatives. Be that as it may, these days, the market has gone aggressive, which is the reason it’s vital to first get your work

...
goodfirms web development company
blog
Attain Growth With Top Digital Marketing Agency at Good Firms: Leading Edge Info Solutions

Listen to this article Helping the clients to garner attention towards their website will soon tap Leading Edge Info Solutions amongst the best web developers in India at GoodFirms. View the Leading Edge Info Solutions’ GoodFirms’ profile to get aware of its standardized process of

...

© 2025 Leading Edge Info Solutions (P) Ltd. All Rights Reserved.

To Top
Book a Slot