AI bot & crawler tracker

Install @xpmetric/ai-crawl so XPmetric can see AI bots and search crawlers that skip JavaScript — including HTTP status codes for 404 ideas.

Most AI bots and search crawlers fetch raw HTML and never execute p.js. The optional @xpmetric/ai-crawl package reports those hits from your server so the Bot Traffic card can show crawl volume, provider trends, and 404 ideas.

Install

npm install @xpmetric/ai-crawl

Copy your site token from Settings → Sites → Get Script (or the Bot Traffic empty state).

Next.js

Middleware cannot see the final App Router status code. Use middleware for hit volume, then re-report 404 from app/not-found.tsx.

middleware.ts

import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
import { trackCrawlerRequest } from "@xpmetric/ai-crawl";

export function middleware(request: NextRequest, event: NextFetchEvent) {
  const requestHeaders = new Headers(request.headers);
  requestHeaders.set("x-xpm-pathname", request.nextUrl.pathname);

  trackCrawlerRequest(request, event, {
    siteToken: "YOUR_SITE_TOKEN",
  });

  return NextResponse.next({ request: { headers: requestHeaders } });
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

app/not-found.tsx

import { headers } from "next/headers";
import { reportCrawlerHit } from "@xpmetric/ai-crawl";

export default async function NotFound() {
  const h = await headers();
  const ua = h.get("user-agent") || "";
  const host = h.get("host") || "localhost";
  const path = h.get("x-xpm-pathname") || "/";
  const proto = h.get("x-forwarded-proto") || "https";

  reportCrawlerHit(
    new Request(`${proto}://${host}${path}`, {
      headers: { "user-agent": ua, referer: h.get("referer") || "" },
    }),
    404,
    undefined,
    { siteToken: "YOUR_SITE_TOKEN" }
  );

  return (
    <main>
      <h1>404</h1>
      <p>Page not found</p>
    </main>
  );
}

Optional: wrap App Router route handlers with withCrawlerRouteHandler to report other status codes.

Express

Status is captured on response finish:

import express from "express";
import { createExpressCrawlerMiddleware } from "@xpmetric/ai-crawl";

const app = express();
app.use(
  createExpressCrawlerMiddleware({
    siteToken: "YOUR_SITE_TOKEN",
  })
);

Cloudflare Workers

import { withCrawlerTracking } from "@xpmetric/ai-crawl";

export default {
  fetch: withCrawlerTracking(
    async (request) => fetch(request),
    { siteToken: "YOUR_SITE_TOKEN" }
  ),
};

Compatibility

POST /api/crawler accepts optional status_code (0–599). Omitting it or sending 0 means “unknown” — older v1 installs keep working. Real status codes power the 404 ideas tab (paths returning 404/410).

Verify

Curl a path on your site with a bot UA, e.g. ChatGPT-User.
Open the site dashboard → Bot Traffic → 404 ideas (for missing paths) or AI answers.