---
title: "Make your site AI-crawlable"
description: "Publish llms.txt, allow major AI bots in robots.txt, and keep a crawlable sitemap so answer engines can find and cite your content."
source: "https://xpmetric.com/docs/ai-crawlable"
---
XPmetric's Bot Traffic card shows **what** AI bots already crawl. This guide covers **what to do** so more of them can find you — the same checklist behind the **AI ready N/4** chip.

## 1. Publish `/llms.txt`

Serve a plain-text index at `https://yoursite.com/llms.txt` that returns **HTTP 200**. Keep it short: what your product is, and links to the pages you want models to read.

```txt
# Example llms.txt
> MySaaS — analytics for indie founders

## Docs
- [Getting started](https://yoursite.com/docs/getting-started): install in 4 steps
- [API overview](https://yoursite.com/docs/api): endpoints and auth

## Optional
- [Pricing](https://yoursite.com/pricing)
```

Tips:

- Prefer absolute HTTPS URLs.
- Link to markdown mirrors (`/docs/foo.md`) if you have them — many agents prefer raw text.
- Re-check the chip after deploy; XPmetric probes your live `/llms.txt` (cached ~24h).

## 2. Don't block major AI bots in `robots.txt`

These agents must **not** see a blanket `Disallow: /` in their User-agent group (or under `*` when no specific group exists):

- `GPTBot` (OpenAI training / crawl)
- `ClaudeBot` (Anthropic)
- `PerplexityBot` (Perplexity)
- `Google-Extended` (Google AI training)

Safe pattern — allow them explicitly even if you restrict others:

```txt
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow:
```

Blocking `/admin` or `/api` for everyone is fine. Blocking `/` for these four agents fails the checklist.

## 3. Keep a crawlable sitemap

Serve `/sitemap.xml` or `/sitemap_index.xml` with a **2xx** response, and make sure bots actually fetch it. XPmetric passes this check when:

1. The sitemap URL exists on your domain, **and**
2. At least one crawler hit that path in the last **30 days** (visible in Bot Traffic → Pages).

Submit the sitemap in Google Search Console / Bing Webmaster if crawls are slow to start.

## 4. Fix bot 404s

Paths that return **404/410** to crawlers waste crawl budget and hide content from answer engines. Open the Bot Traffic **404 ideas** tab, then either:

- Build the missing page, or
- **301** to the canonical URL.

Install [`@xpmetric/ai-crawl`](/docs/bot-tracker) with response-aware reporting so status codes show up — without it, the 404 tab stays empty even when bots hit missing routes.

## Related

- [AI bot & crawler tracker](/docs/bot-tracker) — install the middleware
- [Get started](/docs/getting-started) — human pageview tracking with `p.js`