Middle Mann  /  Free SEO Tools  /  Robots.txt & Sitemap Analyzer

Robots.txt & Sitemap Analyzer

Paste a domain. We fetch robots.txt and sitemap.xml, run 12 structural checks and return a 0 to 100 crawl health score with the exact fix per failing check. No signup.

FAQ

Frequently Asked Questions

What does the analyzer check?

Twelve structural checks across robots.txt and sitemap.xml. robots.txt accessibility, syntax validity, content-type, crawl directive sanity (blanket block detection), sitemap reference. Sitemap accessibility, format (urlset or sitemapindex), URL count, lastmod coverage, URL validity, robots vs sitemap cross-reference, HTTPS consistency. Each returns pass, warn or fail with the specific finding and the exact fix.

Why does crawl health matter?

Google and AI assistants discover your pages by crawling. If robots.txt blocks the wrong path or your sitemap is malformed, pages never get indexed. No index equals no ranking and no citation. Crawl health is the foundation every other SEO signal sits on.

Is this really free?

Yes. The full report shows on screen with no signup. Rate-limited to 10 analyses per IP per day. Optional email field sends you a copy of the report.

What is a sitemap index?

A sitemap index is a sitemap of sitemaps. Large sites split URLs across multiple child sitemaps (one per section, product line, blog category) and reference them all from a top-level sitemapindex file. We follow sitemap indexes one level deep and sample up to 500 URLs across the children.

What is the orphan check?

We cross-reference every URL in your sitemap against the Disallow rules in robots.txt for User-agent: *. A URL that is listed in the sitemap but blocked by robots.txt is a contradiction. You are telling crawlers "please index this" and "do not touch this" at the same time. Pick one.

Will Middle Mann fix the issues for me?

Yes. The full AI SEO and GEO service covers crawl setup, schema overhaul, answer-ready content refactor, entity work and daily rank tracking across every major AI assistant. See the AI SEO service page for details.

About the Tool

What this tool does.

A robots.txt and sitemap analyzer scores the crawl health of your site: whether Google and AI assistants can actually find and index your pages.

Crawl health is the foundation beneath every other SEO signal. If robots.txt has a Disallow: / that blanket-blocks the entire site, nothing gets indexed. If the sitemap is malformed or empty, crawlers do not know what pages exist. If sitemap URLs are blocked by robots.txt rules, you are sending Google contradictory instructions and it just gives up. This tool checks every structural signal that matters, in order, and returns a composite crawl health score with the exact fix per failing check.

All data comes from real HTTP fetches against your origin. We pull robots.txt, parse it as a strict parser would, follow the Sitemap: directive to find your canonical sitemap, parse the XML, follow sitemap indexes one level deep, and sample up to 500 URLs. Then we cross-reference everything: sitemap URLs against robots.txt rules, URL schemes against site scheme, lastmod coverage, URL validity. Zero AI, zero scraping, zero estimation. Fix the fails, re-run, watch the score climb.

12 checks across robots + sitemap.

Real HTTP fetches. Live XML parse.

  • robots.txt AccessibilityHTTP
  • robots.txt SyntaxParser
  • Content-TypeHeader
  • Crawl Directives SanityRules
  • Sitemap ReferenceDirective
  • Sitemap AccessibilityHTTP
  • Sitemap FormatXML
  • URL CountSample
  • lastmod Coverage%
  • URL ValidityParse
  • Robots vs SitemapCross-ref
  • HTTPS ConsistencyScheme
From the Blog

Related Reading

Is Traditional SEO Dead?
29SEP
The State of the Web · 7 min read

Is Traditional SEO Dead?

The short version: no. The long version is more interesting. What actually died, what still works and the four things SEO 2026 needs that SEO 2022 did not.

Read
How Much Does AI SEO Cost?
22SEP
Money & Pricing · 7 min read

How Much Does AI SEO Cost?

Real 2026 pricing for AI SEO and GEO. What tiers actually buy you, hidden costs most agencies will not disclose, and when you can expect to see a return.

Read
More Tools

Try another SEO tool.

Most Middle Mann SEO tools are free, no signup, no email required. We offer paid SEO tools too. Both are self-service and rate-limited. Need higher volume, deeper reporting or something custom? Get ahold of us.

Need a custom SEO engagement?

See AI SEO service
Related

Keep reading.