---
title: "AI bot & crawler tracker"
description: "Install @xpmetric/ai-crawl so XPmetric can see AI bots and search crawlers that skip JavaScript — including HTTP status codes for 404 ideas."
source: "https://xpmetric.com/docs/bot-tracker"
---
Most AI bots and search crawlers fetch raw HTML and never execute `p.js`. The optional `@xpmetric/ai-crawl` package reports those hits from your server so the Bot Traffic card can show crawl volume, provider trends, and **404 ideas**.

## Install

```bash
npm install @xpmetric/ai-crawl
```

Copy your **site token** from **Settings → Sites → Get Script** (or the Bot Traffic empty state).

## Next.js

Middleware cannot see the final App Router status code. Use middleware for hit volume, then re-report **404** from `app/not-found.tsx`.

### middleware.ts

```ts
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
import { trackCrawlerRequest } from "@xpmetric/ai-crawl";

export function middleware(request: NextRequest, event: NextFetchEvent) {
  const requestHeaders = new Headers(request.headers);
  requestHeaders.set("x-xpm-pathname", request.nextUrl.pathname);

  trackCrawlerRequest(request, event, {
    siteToken: "YOUR_SITE_TOKEN",
  });

  return NextResponse.next({ request: { headers: requestHeaders } });
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};
```

### app/not-found.tsx

```ts
import { headers } from "next/headers";
import { reportCrawlerHit } from "@xpmetric/ai-crawl";

export default async function NotFound() {
  const h = await headers();
  const ua = h.get("user-agent") || "";
  const host = h.get("host") || "localhost";
  const path = h.get("x-xpm-pathname") || "/";
  const proto = h.get("x-forwarded-proto") || "https";

  reportCrawlerHit(
    new Request(`${proto}://${host}${path}`, {
      headers: { "user-agent": ua, referer: h.get("referer") || "" },
    }),
    404,
    undefined,
    { siteToken: "YOUR_SITE_TOKEN" }
  );

  return (
    <main>
      <h1>404</h1>
      <p>Page not found</p>
    </main>
  );
}
```

Optional: wrap App Router route handlers with `withCrawlerRouteHandler` to report other status codes.

## Express

Status is captured on response finish:

```ts
import express from "express";
import { createExpressCrawlerMiddleware } from "@xpmetric/ai-crawl";

const app = express();
app.use(
  createExpressCrawlerMiddleware({
    siteToken: "YOUR_SITE_TOKEN",
  })
);
```

## Cloudflare Workers

```ts
import { withCrawlerTracking } from "@xpmetric/ai-crawl";

export default {
  fetch: withCrawlerTracking(
    async (request) => fetch(request),
    { siteToken: "YOUR_SITE_TOKEN" }
  ),
};
```

## Compatibility

`POST /api/crawler` accepts optional `status_code` (0–599). Omitting it or sending `0` means “unknown” — older v1 installs keep working. Real status codes power the **404 ideas** tab (paths returning 404/410).

## Verify

1. Curl a path on your site with a bot UA, e.g. `ChatGPT-User`.
2. Open the site dashboard → Bot Traffic → **404 ideas** (for missing paths) or **AI answers**.