Even though the robots.txt file was invented to tell search engines like google what webpages to not crawl, the robots.txt file can even be used to stage serps for the XML sitemap. This is often supported by Google, Bing, Yahoo and Request. The simplest way to signal to search engines https://rankingseotoolz.com/robots-txt-generator