How to Create a Perfect robots.txt File

Robots.txt is a genuinely tiny file — often just a handful of lines — that carries an outsized amount of responsibility. It's the first thing a well-behaved crawler checks before touching anything else on your site, and it's astonishingly easy to get wrong in a way that's completely invisible until someone finally asks why traffic cratered.

The good news: once you understand the syntax, robots.txt takes about five minutes to get right. Let's walk through it properly, then generate and test one for real.

The basic syntax, explained plainly

A robots.txt file is built from simple rule blocks: a User-agent line naming which bot the rule applies to, followed by Allow or Disallow lines specifying paths. `User-agent: *` means "this applies to every bot." `Disallow: /admin/` means "don't crawl anything under /admin/." That's genuinely most of what you need to know — the complexity people imagine rarely exists in practice.

The mistake that causes the most damage

`Disallow: /` with nothing after the slash blocks your entire site from that user-agent. It's usually left over from a staging environment that got pushed to production without anyone double-checking. This single line, sitting quietly at the top of a file, has cost more sites their rankings than almost any other technical mistake I've seen in a decade of doing this.

What actually belongs in a good robots.txt

Block admin areas, internal search result pages, and staging/testing paths. Don't block CSS or JS files — Google needs those to render your page properly, and blocking them can hurt how your page is evaluated. Always include a Sitemap directive pointing to your XML sitemap, so crawlers know exactly where to find your full URL list.

Build it, then test it — don't skip the second step

Use the Robots.txt Generator to build a clean file from scratch with allow/disallow rules, crawl-delay, and your sitemap URL all in one form. Then — and this step matters — paste the result into the Robots.txt Tester and check specific paths against specific bots before you publish. Writing the file correctly and verifying it behaves correctly are two different steps, and skipping the second one is how mistakes slip through.

Try it free — no signup required.

Generate your robots.txt now

Don't forget AI crawlers

In 2026, robots.txt decisions extend beyond Googlebot. GPTBot, ClaudeBot, PerplexityBot and Google-Extended all respect robots.txt directives, and whether you allow or block them is now a genuine strategic choice, not an afterthought. The GEO Readiness Checklist covers how to think through that decision for your specific site.

Testing before you deploy, every time

Make it a habit to re-check robots.txt any time you migrate hosts, launch a redesign, or push a staging environment live. This is the file most likely to carry over a leftover "block everything" rule from a dev environment, and it's the cheapest five-minute check you can run before a launch that could otherwise cost you weeks of lost crawling.

Quick reference: common directives

Bookmark this table for the rules you'll actually use most often.

Frequently Asked Questions

Does robots.txt stop a page from ever appearing in Google?

Not entirely — a disallowed page can still appear in search results (usually without a description) if other sites link to it. To fully prevent indexing, use a noindex meta tag instead, on a page that's still crawlable.

Where does robots.txt need to live?

At the root of your domain — example.com/robots.txt. Anywhere else, and crawlers won't find it at all.

Can I have different rules for different bots?

Yes — use separate User-agent blocks for each bot you want to target differently, such as one set of rules for Googlebot and another for GPTBot.