Robots.txt Generator

robots.txt is a plain text file at the root of your site, for example example.com/robots.txt, that tells web crawlers which parts of the site they may visit. Well behaved crawlers like Googlebot and Bingbot follow these rules.

Not exactly. Blocking a page stops Google from crawling it, but the URL can still appear in results if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag instead.

Upload it to the top level folder of your domain so it is available at https://yourdomain.com/robots.txt. It will not work in a subfolder. On WordPress, SEO plugins like Yoast and Rank Math let you edit it from the dashboard.

That is a business decision. Blocking crawlers like GPTBot or Google-Extended can stop your content being used to train AI models, but it does not affect normal Google Search rankings. Many publishers block them, while others allow them for visibility in AI answers.

Crawl-delay asks robots to wait a number of seconds between requests to reduce server load. Bing and Yandex respect it, but Google ignores it and manages its own crawl rate. Only use it if your server struggles with bot traffic.