WordPress 5.3 will be implementing a change to it's "discourage search engines from indexing this site" setting.
The discourage search engines from indexing setting in WordPress (found in Settings->Reading) made use of robots.txt, effectively adding in a
"Disallow: /" line which prevented search engines from crawling the site, and whilst this didn't always prevent indexing it did go a long way to ensuring your site wouldn't be listed in Google.
As of WordPress' 5.3 release the popular Content Management system will no longer rely on robots.txt, and will instead use the more reliable meta tag method:
<meta name='robots' content='noindex,nofollow' />
This now allows search engines to crawl the website (which robots.txt discouraged them from doing, but didn't ever fully stop them from doing) but tells them not to index the website, and whilst it's not a 100% guaranteed method to stop your site being indexed, it's better than the previous implementation.
That's a question many people ask - "Why stop my site from being indexed? I've built it so people can find it, after all!" and that's an excellent point - if you're paying for WordPress hosting and spending the time designing your website, writing content, and making it work well for your visitors, it doesn't really make sense to stop it being indexed, does it?
Well - no, not if it's you're live website, however, if you have a clone of your website (a staging site, or a development copy of your website) to work on before pushing changes live, then you most certainly do NOT want that to be indexed - at best, it's going to make the search results in Google look strange with your pages showing up multiple times on different domains/subdomains, and worst case it will actually get you a duplicate content penalty - this means your site will perform poorly in the SERPS.
WordPress has used the robots.txt method pretty much since it's initial release, so why the change? Well, as of September 1st 2019, Google no longer supports robots.txt - any website which is relying solely on robots.txt to avoid being indexed will now be indexed if Google happens to crawl it, if you have a website which only uses robots.txt and do not want it indexed then you need to implement another method as soon as possible.
robots.txt was never something that was "official", it was something most search engines decided to honour, but as things have evolved, it's no longer fit for purpose. Google's official statement (paraphrased) on this is:
“In the interest of maintaining a healthy ecosystem and preparing for potential future open source releases, we’re retiring all code that handles unsupported and unpublished rules (such as noindex) on September 1, 2019.”Google
So in short, if you're relying solely on robots.txt, take action today! WordPress 5.3 will handle this for you, however, it isn't expected to enter early beta until the end of September, with a public release target of week 2 in November 2019.