WordPress 5.3 makes the switch to Robots Meta tags

Posted on 04/09/2019 by Jamie Martin
Developers rejoice! WordPress is switching to use a meta tag for the robots tag, this will mean more control to stop Google indexing development sites! Read more here: https://freshsites.co.uk/wordpress-robots-meta-tags/ #WordPress #Hosting #UK #WebDevelopment #WebDesign

WordPress 5.3 will be implementing a change to it's "discourage search engines from indexing this site" setting.

How did it used to work?

The discourage search engines from indexing setting in WordPress (found in Settings->Reading) made use of robots.txt, effectively adding in a "Disallow: /" line which prevented search engines from crawling the site, and whilst this didn't always prevent indexing it did go a long way to ensuring your site wouldn't be listed in Google.

How will it work in WordPress 5.3 and beyond?

As of WordPress' 5.3 release the popular Content Management system will no longer rely on robots.txt, and will instead use the more reliable meta tag method: <meta name='robots' content='noindex,nofollow' />

This now allows search engines to crawl the website (which robots.txt discouraged them from doing, but didn't ever fully stop them from doing) but tells them not to index the website, and whilst it's not a 100% guaranteed method to stop your site being indexed, it's better than the previous implementation.

Why would I want to stop my website from being indexed?

That's a question many people ask - "Why stop my site from being indexed? I've built it so people can find it, after all!" and that's an excellent point - if you're paying for WordPress hosting and spending the time designing your website, writing content, and making it work well for your visitors, it doesn't really make sense to stop it being indexed, does it?

Well - no, not if it's you're live website, however, if you have a clone of your website (a staging site, or a development copy of your website) to work on before pushing changes live, then you most certainly do NOT want that to be indexed - at best, it's going to make the search results in Google look strange with your pages showing up multiple times on different domains/subdomains, and worst case it will actually get you a duplicate content penalty - this means your site will perform poorly in the SERPS.

Why the change now?

WordPress has used the robots.txt method pretty much since it's initial release, so why the change? Well, as of September 1st 2019, Google no longer supports robots.txt - any website which is relying solely on robots.txt to avoid being indexed will now be indexed if Google happens to crawl it, if you have a website which only uses robots.txt and do not want it indexed then you need to implement another method as soon as possible.

Why did Google stop supporting robots.txt?

robots.txt was never something that was "official", it was something most search engines decided to honour, but as things have evolved, it's no longer fit for purpose. Google's official statement (paraphrased) on this is:

“In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we’re retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019.”

Google

So in short, if you're relying solely on robots.txt, take action today! WordPress 5.3 will handle this for you, however, it isn't expected to enter early beta until the end of September, with a public release target of week 2 in November 2019.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

© Copyright 2019 FreshSites Limited. Registered in England and Wales no. 7812509
Website Design by Pyxal
WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites WordPress 5.3 makes the switch to Robots Meta tags | FreshSites