XPmetric

Make your site AI-crawlable

Publish llms.txt, allow major AI bots in robots.txt, and keep a crawlable sitemap so answer engines can find and cite your content.

XPmetric's Bot Traffic card shows what AI bots already crawl. This guide covers what to do so more of them can find you — the same checklist behind the AI ready N/4 chip.

1. Publish /llms.txt

Serve a plain-text index at https://yoursite.com/llms.txt that returns HTTP 200. Keep it short: what your product is, and links to the pages you want models to read.

# Example llms.txt
> MySaaS — analytics for indie founders

## Docs
- [Getting started](https://yoursite.com/docs/getting-started): install in 4 steps
- [API overview](https://yoursite.com/docs/api): endpoints and auth

## Optional
- [Pricing](https://yoursite.com/pricing)

Tips:

  • Prefer absolute HTTPS URLs.
  • Link to markdown mirrors (/docs/foo.md) if you have them — many agents prefer raw text.
  • Re-check the chip after deploy; XPmetric probes your live /llms.txt (cached ~24h).

2. Don't block major AI bots in robots.txt

These agents must not see a blanket Disallow: / in their User-agent group (or under * when no specific group exists):

  • GPTBot (OpenAI training / crawl)
  • ClaudeBot (Anthropic)
  • PerplexityBot (Perplexity)
  • Google-Extended (Google AI training)

Safe pattern — allow them explicitly even if you restrict others:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow:

Blocking /admin or /api for everyone is fine. Blocking / for these four agents fails the checklist.

3. Keep a crawlable sitemap

Serve /sitemap.xml or /sitemap_index.xml with a 2xx response, and make sure bots actually fetch it. XPmetric passes this check when:

  1. The sitemap URL exists on your domain, and
  2. At least one crawler hit that path in the last 30 days (visible in Bot Traffic → Pages).

Submit the sitemap in Google Search Console / Bing Webmaster if crawls are slow to start.

4. Fix bot 404s

Paths that return 404/410 to crawlers waste crawl budget and hide content from answer engines. Open the Bot Traffic 404 ideas tab, then either:

  • Build the missing page, or
  • 301 to the canonical URL.

Install @xpmetric/ai-crawl with response-aware reporting so status codes show up — without it, the 404 tab stays empty even when bots hit missing routes.

Related