Free Robots.txt Generator
Build a valid robots.txt file in seconds. Choose a preset, configure bot rules and blocked paths, add your sitemap URL - then copy or download, ready to upload.
- Pre-built presets for WordPress, e-commerce & more
- Configure per-bot Allow, Disallow & Crawl-delay rules
- Add sitemap URL directly into the file
- One-click copy or download as robots.txt
80+ brands managed
100%
Free - no sign-up
Valid
Google-accepted format
Quick Presets
Bot Rules
Use * for all bots, or specify e.g. Googlebot
Press Enter or comma to add · Backspace to remove last
Press Enter or comma to add · Backspace to remove last
Asks bots to wait N seconds between requests. Leave blank to omit.
Sitemap URLs (optional)
Tells search engines where your sitemap lives.
Control What Bots Can See
A robots.txt file tells search engines and bots which pages to crawl - and which to leave alone.
Per-Bot Configuration
Set different Allow, Disallow and Crawl-delay rules per bot. Block AI scrapers like GPTBot while keeping Google fully open.
One-Click Presets
WordPress, e-commerce, block-all and allow-all presets let you generate a sensible robots.txt in seconds with no technical knowledge.
Protect Private Areas
Easily block admin panels, API endpoints, checkout pages and private directories from being indexed by search engines.
Sitemap Declaration
Add your sitemap.xml URL directly in the robots.txt file - the recommended way to help crawlers discover all your pages.
Copy or Download
One-click copy or download as robots.txt - ready to upload to your website root and verified in Google Search Console.
100% Free - Always
No account, no limits, no watermarks. Generate as many robots.txt files as you need, completely free forever.
robots.txt Ready in 4 Simple Steps
From blank to upload-ready in under 2 minutes.
Choose a Preset
Select a starting point - Allow All, Block All, WordPress, e-commerce or Standard SEO - then customise from there.
Configure Bot Rules
Add User-agent rules for specific bots. Set which paths are allowed, which are blocked, and optionally set a crawl delay.
Add Your Sitemap
Paste your sitemap.xml URL to include a Sitemap directive - the best way to help Google discover all your pages.
Upload & Verify
Download robots.txt, upload it to your website root, then verify it's working in Google Search Console under Settings.
Choose a Preset
Select a starting point - Allow All, Block All, WordPress, e-commerce or Standard SEO - then customise from there.
Configure Bot Rules
Add User-agent rules for specific bots. Set which paths are allowed, which are blocked, and optionally set a crawl delay.
Add Your Sitemap
Paste your sitemap.xml URL to include a Sitemap directive - the best way to help Google discover all your pages.
Upload & Verify
Download robots.txt, upload it to your website root, then verify it's working in Google Search Console under Settings.
Why Every Website Needs One
A robots.txt file sits at the root of your website and gives instructions to web crawlers - telling them which pages and directories they're allowed to access. It's not a security measure (bots can choose to ignore it), but all major search engines respect it, making it an essential part of any technical SEO setup.
Generate Mine NowControls Crawler Access
Robots.txt tells Googlebot, Bingbot and other crawlers exactly which pages they're allowed to visit - protecting private or duplicate content from being indexed.
Manages Crawl Budget
Large sites have a limited crawl budget. Blocking unimportant pages (admin, filters, duplicate URLs) ensures crawlers focus on your most valuable content.
Blocks AI Scrapers
Add rules for GPTBot, CCBot and anthropic-ai to prevent AI companies from using your content to train their models without permission.
An SEO Foundation File
Every professional website needs a robots.txt. It's one of the first files search engines check and signals that your site is technically well-maintained.
robots.txt Questions Answered
A robots.txt file is a plain text file placed at the root of your website (e.g. yourdomain.com/robots.txt) that gives instructions to web crawlers about which pages or sections they're allowed to access. All major search engines - including Google, Bing and Yandex - respect robots.txt directives.
Blocking a page in robots.txt prevents crawlers from visiting it, but doesn't guarantee it won't be indexed. If other sites link to a blocked page, Google may still index it based on those links. To fully prevent indexing, use a 'noindex' meta tag or HTTP header on the page itself.
Disallow tells a bot not to crawl a specific path. Allow overrides a broader Disallow rule for a specific sub-path. For example, you can Disallow /wp-admin/ but Allow /wp-admin/admin-ajax.php to keep AJAX requests working while blocking the dashboard.
Most legitimate bots (Googlebot, Bingbot) should be allowed to crawl your public content. You might want to block specific bots like AI scrapers (GPTBot, CCBot) if you don't want your content used for model training. Never block Googlebot from pages you want indexed.
Crawl-delay asks a bot to wait N seconds between requests to reduce server load. Note: Googlebot ignores Crawl-delay - you should set crawl rate for Google in Search Console instead. Crawl-delay is respected by Bingbot and some other crawlers.
Upload robots.txt to the root directory of your website so it's accessible at yourdomain.com/robots.txt. For most hosting providers, this means placing it in the /public_html or /www directory. You can then verify it in Google Search Console under Settings → robots.txt.
robots.txt Is Just the Start.
Let's Build Your Full SEO Strategy.
Our SEO team handles everything from technical audits and on-page optimisation to content strategy and link building - turning organic search into your biggest growth channel.