How to Use the Robots.txt Generator
- Add a rule block for a user-agent (use "*" to target all crawlers).
- List paths to disallow or allow under that user-agent.
- Optionally add your sitemap URL.
- Copy the generated robots.txt content or download it.
How robots.txt works
robots.txt is a plain-text file placed at the root of a domain (e.g. example.com/robots.txt) that well-behaved crawlers check before crawling a site. It uses simple `User-agent`, `Disallow`, and `Allow` directives to specify which paths a given crawler should avoid or is permitted to access.
Important limitations
robots.txt is a voluntary convention — legitimate search engines respect it, but it is not a security or access-control mechanism, and malicious crawlers can simply ignore it. Never rely on robots.txt to keep sensitive content private; use proper authentication instead. Also note that disallowing a page in robots.txt does not guarantee it won't appear in search results if it's linked from elsewhere — use a noindex meta tag for that purpose.
Example
Allow all crawlers, add sitemap
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
Tips
- Never use robots.txt as a way to hide sensitive or private content — it is publicly readable and non-binding.
- Always include a Sitemap directive pointing search engines to your sitemap.xml.
Frequently Asked Questions
Does robots.txt keep pages out of Google entirely?
Not necessarily. It stops crawling of that path, but a URL can still appear in search results (without a description) if it's linked from elsewhere. Use a noindex meta tag to reliably prevent indexing.
Do all crawlers respect robots.txt?
Reputable search engines do, but robots.txt is voluntary and cannot force compliance — it is not a security mechanism.