Robots.txt Generator
Create a robots.txt file with user-agent, disallow and allow rules, crawl delay, and sitemap. Free and private.
Independently verified for accuracy
Calculator by Toolsloft ↗Build a robots.txt file that tells search engine crawlers which parts of your site they may and may not visit. Set the user-agent, list the paths to disallow or allow, and add a sitemap line, then copy the result to your site root. Everything is generated in your browser.
How this is calculated
Output follows the robots exclusion format: a User-agent line, one Disallow or Allow line per path, an optional Crawl-delay when a number is given, and a Sitemap line when a URL is set. When no rules are entered it emits an empty Disallow, which allows crawling of the whole site.
How to use
- Set the user-agent, or leave it blank to target all crawlers.
- List the paths to disallow, one per line, and any paths to allow.
- Add a crawl delay and sitemap URL if you use them, then copy the file to your site root.
Examples
- Block folders:
/admin/ and /private/ become two Disallow lines - Allow all:
no rules produces User-agent: * and an empty Disallow
FAQ
- Where does the robots.txt file go?
- It must sit at the root of your domain, for example https://example.com/robots.txt. Crawlers only read it from that location.
- What does an empty Disallow mean?
- An empty Disallow line allows crawling of the entire site. It is the standard way to say there is nothing to block.
- Do all crawlers follow robots.txt?
- Major search engines respect it, but it is a convention, not a security control. Do not rely on it to hide sensitive pages.