AI bot & crawler tracker

Most AI bots and search crawlers fetch raw HTML and never execute p.js. Server reporting shows those visits in Bot Traffic, including provider trends and 404 ideas. On WordPress, one XPmetric Bot Tracker plugin reports likely crawlers with the actual HTTP status and can optionally add browser tracking for human pageviews. For other server apps, use @xpmetric/ai-crawl.

WordPress plugin

The plugin reports likely crawler requests from your server with the actual HTTP status, so pages that return 404 can appear in 404 ideas. It can also inject XPmetric's browser script for human pageviews without editing theme files.

Download XPmetric Bot Tracker 1.1.0 and verify it with the SHA-256 checksum. In WordPress admin, go to Plugins → Add New Plugin → Upload Plugin, install the ZIP, and activate it.

Open Settings → XPmetric Bot Tracker. Copy the site token from XPmetric Settings → Sites → Get Script, paste it, turn on Enable bot tracking, and save. Click Send test hit. A successful result says Test hit accepted by XPmetric. In Bot Traffic, look for the synthetic path /xpmetric-test, usually within a minute.

To include human pageviews, open Human analytics tracking and check Inject tracking script in the site head. Select a Tracking mode that matches that site's setting in XPmetric Settings → Sites → Site. Cookieless loads https://xpmetric.com/p.js; Cookie loads https://xpmetric.com/p.cookie.js. The plugin displays the selected script URL and checks active WPCode snippets and the rendered page before injection. If it finds an XPmetric script, or cannot verify that the page is clear, it refuses to enable injection. Remove or disable the existing WPCode or theme script before switching to plugin injection, or leave plugin injection off and keep your current script placement. Do not load the same script twice.

Cookie mode adds the script automatically. If your site uses a consent tool, configure it to load the script only after analytics consent, or leave plugin injection off and let the consent tool add it. XPmetric does not include a consent tool. Depending on where your visitors are, you may need a consent banner before cookie-mode tracking.

The test result shows the HTTP status and API response. A 403 means the saved site token was rejected. A 402 can mean the subscription or trial is inactive, so XPmetric has paused collection. Collection needs an active subscription or trial and is not restricted by a particular plan. A 400 means the test crawler was not accepted. Connection problems are shown separately. The test works while regular bot tracking is disabled.

WordPress page cache fallback

The regular plugin cannot recover a page that an advanced-cache.php drop-in has already served and exited before regular plugins load. The settings page recognizes common cache names, including WP Rocket, W3 Total Cache, LiteSpeed Cache, WP Super Cache, Batcache, and Cache Enabler, and links to this fallback.

For a PHP page cache that exits early, add these two lines at the top of wp-config.php, before WP_CACHE is defined:

if ( ! defined( 'ABSPATH' ) ) define( 'ABSPATH', __DIR__ . '/' );
if ( file_exists( ABSPATH . 'wp-content/plugins/xpmetric-bot-tracker/xpmetric-bot-tracker-early.php' ) ) require_once ABSPATH . 'wp-content/plugins/xpmetric-bot-tracker/xpmetric-bot-tracker-early.php';

Keep the ABSPATH initialization first so the early file can load before WordPress defines it. The example assumes wp-config.php is in the WordPress install directory and the plugin is in wp-content/plugins/xpmetric-bot-tracker. Adjust the paths if your WordPress or plugin directory is custom. The file-existence check means removing the plugin does not break the site.

The early callback reads the saved enable switch, token, and endpoint from WordPress's options table using the database constants and table prefix in wp-config.php. It writes no token file or settings snapshot. PHP's MySQLi extension and access to the options table are required. If regular WordPress loading continues, its reporter sends the hit and the early callback skips it, preventing duplicates. Disabling tracking or deactivating the plugin stops reporting, including for cached requests.

This fallback covers PHP requests only. A CDN, a cache at the LiteSpeed server level, or a rewrite rule serving a Disk Enhanced cache file can respond without running PHP, so this snippet cannot report those hits. Configure that cache to bypass likely crawlers or use an edge reporting path such as Cloudflare Workers.

Install

npm install @xpmetric/ai-crawl

Copy your site token from Settings → Sites → Get Script (or the Bot Traffic empty state).

Next.js

Middleware cannot see the final App Router status code. Use middleware for hit volume, then re-report 404 from app/not-found.tsx.

middleware.ts

import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
import { trackCrawlerRequest } from "@xpmetric/ai-crawl";

export function middleware(request: NextRequest, event: NextFetchEvent) {
  const requestHeaders = new Headers(request.headers);
  requestHeaders.set("x-xpm-pathname", request.nextUrl.pathname);

  trackCrawlerRequest(request, event, {
    siteToken: "YOUR_SITE_TOKEN",
  });

  return NextResponse.next({ request: { headers: requestHeaders } });
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

app/not-found.tsx

import { headers } from "next/headers";
import { reportCrawlerHit } from "@xpmetric/ai-crawl";

export default async function NotFound() {
  const h = await headers();
  const ua = h.get("user-agent") || "";
  const host = h.get("host") || "localhost";
  const path = h.get("x-xpm-pathname") || "/";
  const proto = h.get("x-forwarded-proto") || "https";

  reportCrawlerHit(
    new Request(`${proto}://${host}${path}`, {
      headers: { "user-agent": ua, referer: h.get("referer") || "" },
    }),
    404,
    undefined,
    { siteToken: "YOUR_SITE_TOKEN" }
  );

  return (
    <main>
      <h1>404</h1>
      <p>Page not found</p>
    </main>
  );
}

Optional: wrap App Router route handlers with withCrawlerRouteHandler to report other status codes.

Express

Status is captured on response finish:

import express from "express";
import { createExpressCrawlerMiddleware } from "@xpmetric/ai-crawl";

const app = express();
app.use(
  createExpressCrawlerMiddleware({
    siteToken: "YOUR_SITE_TOKEN",
  })
);

Cloudflare Workers

import { withCrawlerTracking } from "@xpmetric/ai-crawl";

export default {
  fetch: withCrawlerTracking(
    async (request) => fetch(request),
    { siteToken: "YOUR_SITE_TOKEN" }
  ),
};

Compatibility

POST /api/crawler accepts optional status_code (0–599). Omitting it or sending 0 means “unknown” — older v1 installs keep working. Real status codes power the 404 ideas tab (paths returning 404/410).

Verify

1. Curl a path on your site with a bot UA, e.g. ChatGPT-User.
2. Open the site dashboard → Bot Traffic → 404 ideas (for missing paths) or AI answers.