Robots.txt tester
Robots.txt tester
Fetch, parse and analyze any website's robots.txt. Automatic detection of blocking rules, declared sitemaps and per-bot directives.
What is robots.txt for?
The robots.txt file, placed at your domain root, tells crawlers (Googlebot, Bingbot, etc.) which site parts they may or may not explore. It's your first line of defense to control crawl budget.
A bad rule can accidentally deindex critical pages. A good robots.txt also includes your XML sitemap URLs to help discovery.
Directive syntax
User-agent:— Specifies which bot the following block applies to (e.g. Googlebot, * for all).Disallow:— URL or folder forbidden from crawling.Allow:— Exception to a parent Disallow rule.Sitemap:— Absolute URL of an XML sitemap to consider.Crawl-delay:— Seconds to wait between requests (ignored by Googlebot).
Frequently asked questions
No, everything is allowed by default. But having one lets you guide bots and declare your sitemaps.
No, only crawling. A page blocked in robots.txt can still appear in results if it has backlinks. To deindex, use noindex meta tag.
Yes. Add specific blocks: User-agent: GPTBot, User-agent: ClaudeBot, User-agent: CCBot, User-agent: Google-Extended, then Disallow: /
Strongly recommended. It's the simplest way for all bots to discover your sitemaps without Search Console or Bing Webmaster Tools.
Related tools
More free SEO tools to optimize your content.
Take it further
Create a free account
Your robots.txt is clean. Who's writing your 10 articles this week?
A technically perfect site without consistent content doesn't rank. Skoatch writes and publishes SEO articles continuously, while you sleep.
SEO articles in 2 minutes
AI images included
Auto WordPress publishing