About robots.txt generator
A robots.txt file sits at the root of a site and tells crawlers which areas they may visit. It guides crawling but is not a security feature.
Key directives
- User-agent: which crawler the rules apply to. * means all.
- Disallow: a path the crawler should not fetch.
- Allow: an exception inside a disallowed path.
- Sitemap: the full URL of your XML sitemap.
Common mistakes
- Writing Disallow: / by accident, which blocks the whole site.
- Using robots.txt to hide private pages. The file itself is public and blocked pages can still be listed if linked elsewhere. Use authentication or noindex instead.
- Blocking CSS or JavaScript files that search engines need to render your pages.
Upload location
The file must be available at yoursite.com/robots.txt. Subdomains need their own file.
A worked example
A common file is: User-agent: *, Disallow: /api/, Allow: /, then Sitemap: https://example.com/sitemap.xml. It lets all crawlers visit the site except the /api/ area and tells them where the sitemap is.
Related controls
- Use a noindex meta tag to keep a page out of search results. robots.txt only controls crawling.
- Test your file in Search Console after you upload it.
- Name AI crawlers separately if you want different rules for them.
