QuilllyRobots.txt Generator
Generate a robots.txt file to control how search engines crawl your site. Free, instant.
robots.txt
User-agent: * Allow: / Disallow: /admin Sitemap: https://yourdomain.com/sitemap.xml
What is robots.txt?
Robots.txt is a text file placed at the root of your website that tells search engine crawlers which pages they can and cannot access. It is part of the Robots Exclusion Protocol and is respected by all major search engines including Google, Bing, and Yahoo.
Robots.txt directives
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Which crawler the rules apply to | User-agent: * |
| Allow | Explicitly allow crawling a path | Allow: /blog/ |
| Disallow | Block crawling a path | Disallow: /admin/ |
| Sitemap | Location of your XML sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Seconds between requests (Bing) | Crawl-delay: 10 |
Frequently Asked Questions
Where do I put robots.txt?
Place the file at the root of your domain: https://yourdomain.com/robots.txt. It must be accessible at this exact URL for crawlers to find it.
Does robots.txt block pages from appearing in Google?
No. Disallowing a page prevents crawling but not indexing. Google may still index the URL if other pages link to it. To prevent indexing, use a noindex meta tag instead.
Should I block AI bots with robots.txt?
If you don't want AI companies using your content for training, you can add specific user-agent rules for GPTBot, ChatGPT-User, and CCBot. Use the "Block AI Bots" preset above as a starting point.
Related free tools
Browse all toolsPaste any URL and find out why it isn’t ranking: meta tags, content quality, AI-agent readiness, robots/sitemap, and Domain Rating — with a copy-paste diagnosis.
Check any domain’s authority score (DR by Ahrefs) free, see what it means for your rankings, and learn the fastest free ways to grow it.
Notify Bing, Yandex, Seznam and Naver about new or updated pages instantly. Generate a key, verify it, and submit up to 100 URLs.
Generate a spec-correct llms.txt (with sitemap prefill) or validate an existing one — so ChatGPT, Claude and Perplexity can read and cite your site.