Every search engine crawler reads your site’s robots.txt file before it starts crawling. That small text file tells bots which sections they may visit and which are off-limits — admin panels, staging areas, thank-you pages, faceted search URLs that would waste your crawl budget. Writing it by hand is error-prone: one misplaced slash can block your entire site from Google. Our free Robots.txt Generator lets you build the file visually, rule by rule, and download a clean, correctly formatted result.
It is designed for website owners and SEOs setting up a new site or auditing an old one. Add user-agent groups (or use * for all bots), choose Allow or Disallow for each path, and optionally point crawlers at your XML sitemap. The live preview shows exactly what will be written to the file, so you can spot mistakes before they go live. The tool is free, needs no signup, and runs entirely in your browser — just remember to upload the downloaded file to your domain root as robots.txt afterwards.
🤖 Robots.txt Generator
Build a robots.txt file from crawl rules you select, then download it.
No rules yet — add at least one above.
How to use
- Enter a user-agent (use
*for all crawlers) and the path to allow or block, e.g./admin/. - Choose Allow or Disallow and click Add Rule. Repeat for every rule.
- Remove any mistaken rule with its delete button.
- Optionally add your sitemap URL so crawlers discover it faster.
- Click Generate robots.txt and check the preview carefully.
- Download the file and upload it to the root of your domain (https://example.com/robots.txt).
FAQs
Can robots.txt keep a page out of Google completely?
Not reliably. Disallow stops compliant crawlers from visiting, but if other sites link to the URL, Google may still index it without content. For pages that must stay private, use password protection or a noindex meta tag instead — robots.txt is a crawling guide, not a security tool.
What is the difference between Allow and Disallow?
Disallow blocks crawlers from the listed path (and everything under it); Allow explicitly permits a path, which is useful to carve out exceptions inside a disallowed directory, e.g. disallow /images/ but allow /images/public/.
Where do I put the downloaded file?
Upload it to the root of your domain so it is reachable at https://yourdomain.com/robots.txt. Search engines only check that one location — a robots.txt in a subfolder is ignored.