Write crawler rules with presets, including one that blocks AI training crawlers.
robots.txt asks crawlers not to visit a path. It does not hide anything, so never use it to protect private pages.
User-agent: * Disallow: /admin/ Disallow: /api/ Sitemap: https://example.com/sitemap.xml
Allow everything is the right default for most sites. The typical preset blocks the paths that usually should not be crawled, and there is one for blocking AI training crawlers.
Each group targets a user-agent. Put one path per line under Allow or Disallow. A more specific group for a named bot overrides the wildcard group for that bot.
The sitemap line helps crawlers find your URLs. Download the file and put it at the root of your domain, since robots.txt is only read from there.
At the root of the domain, so example.com/robots.txt. Crawlers only look there, and a file in a subdirectory is ignored completely.
No, and this is the most common misunderstanding. It asks well-behaved crawlers not to fetch a path. The file is public, so listing a secret path there advertises it. Use authentication for anything private.
Not reliably. A blocked page can still be indexed from external links, showing with no description. To remove a page use a noindex tag, and make sure the page is not blocked so the tag can be read.
You can ask them not to crawl, and the major ones such as GPTBot and CCBot document that they honour robots.txt. The preset here adds those rules while leaving search crawlers allowed.
Crawlers assume everything is allowed, which is fine for most sites. A missing file is not an error.
A crawler uses the single most specific group that names it, and ignores the rest. So a Googlebot group completely replaces the wildcard group for Googlebot rather than adding to it.
Google ignores it. Bing and some others respect it. If Google is crawling too aggressively, adjust the rate in Search Console instead.
Yes, completely free with no sign-up.
Turn a list of URLs into a valid XML sitemap you can copy or download.
Open ToolBuild hreflang tags or sitemap alternates, with a check for the mistakes that silently break them.
Open ToolFree website SEO analyzer. Audit any site for SEO, security, performance, accessibility and mobile-friendliness, with actionable fixes.
Open Tool