See which AI bots crawl your site

Which bots does it recognize?

• AI answer fetch

  answer_fetchDashboard tab: AI answers

  An AI assistant fetching your page while a person waits for an answer.

  • ChatGPT-User
  • Claude-Web
  • Perplexity-User
  • Google-NotebookLM
  • Grok-DeepSearch
  • xAI-SearchBot

• Search indexer

  search_indexDashboard tab: Indexing

  Search engines and AI search indexes building their list of pages.

  • OAI-SearchBot
  • PerplexityBot
  • Googlebot
  • Google-InspectionTool
  • Storebot-Google
  • Bingbot
  • DuckDuckBot
  • YandexRenderResourcesBot
  • YandexBot
  • Sogou web spider
  • Baiduspider
  • Applebot
  • Slurp

• AI training

  trainingDashboard tab: Training

  Crawlers that collect pages to train AI models.

  • GPTBot
  • ClaudeBot
  • anthropic-ai
  • Google-Extended
  • GoogleOther
  • Google-CloudVertexBot
  • CCBot
  • meta-externalagent
  • FacebookBot
  • Bytespider
  • Amazonbot
  • cohere-ai
  • Diffbot

• AI crawler

  ai_crawlerDashboard tab: Training

  Other known AI crawlers. The dashboard shows them on the Training tab.

  • GrokBot
  • xAI-Bot
  • YouBot
  • AI2Bot
  • ImagesiftBot
  • omgili
  • Timpibot

A crawler whose user agent is not in the catalog is not reported as a bot.

What do you see in the dashboard?

• Bot traffic against human pageviews

  One ratio at the top of the card, with the bot hits and human pageviews behind it.

• AI answers, Indexing and Training

  A trend per class over the period you pick, and the providers ranked by hits, such as OpenAI, Anthropic and Perplexity.

• AI citation funnel

  On the AI answers tab, the pages an assistant's bots crawled next to the visitors that arrived from that assistant: ChatGPT, Perplexity, Gemini or Claude.

• Pages

  Which of your pages the bots read, with a search over paths.

• 404 ideas

  Paths bots requested that returned a 404 or 410, with hit counts and the bots that asked. Build the page or redirect it.

• AI ready

  A chip that checks four things: an llms.txt file, robots.txt not blocking the major AI bots, a sitemap that bots crawl, and no bot 404s. The AI-crawlable guide walks through each one.

How do I install it?

npm install @xpmetric/ai-crawl

Next.js

import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
import { trackCrawlerRequest } from "@xpmetric/ai-crawl";

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const requestHeaders = new Headers(request.headers);
  requestHeaders.set("x-xpm-pathname", request.nextUrl.pathname);

  trackCrawlerRequest(request, event, {
    siteToken: "YOUR_SITE_TOKEN",
  });

  return NextResponse.next({ request: { headers: requestHeaders } });
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

On Next.js 15 or earlier, name the file middleware.ts and export middleware. Proxy cannot see the final status of an App Router page, so it reports volume only. To get real 404s, also report from app/not-found.tsx: the snippet is in the guide.

Express

import express from "express";
import { createExpressCrawlerMiddleware } from "@xpmetric/ai-crawl";

const app = express();
app.use(
  createExpressCrawlerMiddleware({
    siteToken: "YOUR_SITE_TOKEN",
  })
);

The status code is captured when the response finishes.

Cloudflare Workers

import { withCrawlerTracking } from "@xpmetric/ai-crawl";

export default {
  fetch: withCrawlerTracking(
    async (request) => fetch(request),
    { siteToken: "YOUR_SITE_TOKEN" }
  ),
};

WordPress

1. 1
   Download XPmetric Bot Tracker 1.2.0 and check it against its SHA-256 checksum.
2. 2
   In WordPress admin, go to Plugins → Add New Plugin → Upload Plugin, install the ZIP and activate it.
3. 3
   Open Settings → XPmetric Bot Tracker, paste your site token, turn on Enable bot tracking and save.
4. 4
   Click Send test hit. A successful result says the hit was accepted, and the path /xpmetric-test appears in Bot Traffic, usually within a minute.

The plugin can also add the browser script for human pageviews. It is checked by default for a new install, and the plugin refuses to add a second copy if it finds one. A page cache that answers before WordPress loads needs an early-load fallback, described in the guide and the WordPress install page.

Check that it works

curl -A "ChatGPT-User" https://yoursite.com/

Then open Bot Traffic on your site dashboard and look at the AI answers tab.

What are the limits?

• Detection is by user agent

  Labels are unverified: a client can send any user agent, and a bot that sends a browser user agent is not recognized.

• Only requests your code sees

  A page answered by a CDN or a page cache without running your code is not reported. Cloudflare Workers can report at the edge.

• Next.js needs one extra file for 404s

  Without the not-found.tsx report, the 404 ideas tab stays empty even when bots hit missing routes.

• A per-site cap

  XPmetric records up to 200 bot hits a minute for a site. Above that, the card says how many were turned away and treats its counts as a floor.

• Needs an active trial or subscription

  Bot tracking is not tied to a particular plan, but collection pauses when the trial or subscription is inactive.

Common questions

Where to go next

Docs: Bot tracker, Make your site AI-crawlable, WordPress, Pricing.