Tool
Robots.txt Generator
Generate robots.txt files for search engine crawlers.
Marketing Tools
What this tool does
Build a robots.txt file to guide search engine crawlers on which paths to index or ignore.
How to use
- Set user-agent (default *).
- Configure Allow and optional Disallow paths.
- Add sitemap URL and copy the file.
Examples
Standard site
Input: Allow: /, Disallow: /admin/, Sitemap: https://example.com/sitemap.xml
Output: Valid robots.txt file
Common pattern for public sites with admin areas blocked.
Common use cases
- Launch a new website
- Block staging or admin paths
- Add sitemap reference for crawlers
FAQ
Does robots.txt guarantee pages won't be indexed?
No. It guides crawlers but pages with external links may still appear.
Where do I put robots.txt?
Upload to your site root: https://yoursite.com/robots.txt
Can I block specific bots?
Set User-agent to the bot name instead of * for bot-specific rules.
Related tools
Last reviewed: 2026-05-23