Robots.txt Generator and Tester
A robots.txt is easy to write badly. This generator produces a correct file and a tester that shows the effective result for any path on your site.
How this tool works
A robots.txt file is a set of rules for crawlers, and it is the first thing a crawler reads. The format is deceptively simple, which is where the mistakes come from: a Disallow line with a path that does not exist blocks nothing, a rule for User-agent * that is followed by a more specific rule does not merge, and a sitemap line that is not a full URL is ignored.This generator produces a valid file from the choices that matter: which paths to disallow, which agents to allow or block, and where the sitemap lives. It also reports what the file will actually do, because a rule that looks restrictive and is not is worse than no rule at all. The path matching is prefix based, so Disallow: /admin blocks /admin, /admin/ and /admin/x, and there is no way to express a path that ends in a specific character without a wildcard.
When to use it
- Start a new site or a new section and the rules are obvious
- Audit an existing file and see which rules are doing nothing
- Block an internal search or an admin area without blocking the rest of the site
- Check that the sitemap line is a full URL, which is the most common error
Frequently asked questions
What is the most common robots.txt mistake?
A Disallow path that does not exist, which blocks nothing, and a Sitemap line that is not a full URL, which is ignored.
Does Disallow: /admin also block /admin/xyz?
Yes. Path matching is prefix based, so it blocks the path and everything under it.
Can I block a search engine from my site?
You can ask, in robots.txt, and most of the major engines honour it. Note that it is a request rather than a control, and a crawler that ignores it will still fetch the page.
Does robots.txt affect ranking?
Not directly. It controls crawling, not indexing, and a page that is blocked cannot be crawled, so it cannot be indexed or ranked.
Should I disallow my admin area?
Usually yes, though it is not a security control. The directory should be protected by authentication, because robots.txt does not keep anyone out.
How often do crawlers read robots.txt?
Google caches it for roughly 24 hours, and other engines cache it for similar periods. A change does not take effect instantly.
What is the difference between Allow and Disallow?
They are the same mechanism with opposite polarity, and the most specific matching rule wins. Allow is usually used to re-permit a path inside a broader Disallow.