# pvuv.ai crawler sync — instructions for an AI coding assistant

The pvuv.ai script only sees visitors whose browser runs JavaScript. AI assistants
(ChatGPT-User, OAI-SearchBot, GPTBot, Claude-User, PerplexityBot…) and search
engines (Googlebot, Bingbot…) fetch the HTML and leave, so they are invisible to
it. This guide adds a few lines to the site's **server** so it reports those
fetches to pvuv.ai. pvuv.ai checks each one against the crawler operator's
published IP ranges (a fake user agent is flagged) and shows which pages AI
assistants read and how many visitors they send afterwards.

Site id: `YOUR_SITE_ID`

## What to build

On every request the site's server answers:

1. If the method is GET or HEAD, the path is a page (not `/api/…`, not `/_…`,
   not a static asset — `.html`, `.txt`, `.xml`, `.md` and extension-less
   paths count as pages) and the User-Agent matches
   `/chatgpt-user|oai-searchbot|gptbot|perplexity-user|perplexitybot|claude-user|claude-searchbot|claudebot|bingbot|googlebot|applebot|duckduckbot|duckassistbot|yandexbot|meta-externalagent|bytespider/i`,
2. then **after or alongside** the response — never delaying it, never failing it —
   POST the hit to pvuv.ai.

### The API

`POST https://api.pvuv.ai/v1/sites/YOUR_SITE_ID/crawler_hits`
Headers: `Authorization: Bearer <PVUV_INGEST_TOKEN>`, `Content-Type: application/json`

```json
{ "hits": [ { "ua": "Mozilla/5.0 … ChatGPT-User/1.0", "path": "/guide/setup", "ip": "20.171.1.2", "status": 200, "ts": 1761465600000 } ] }
```

| field | required | |
|---|---|---|
| `ua` | yes | the request's User-Agent |
| `path` | yes | path (query string is dropped) |
| `ip` | recommended | the client address — used once to verify the crawler, never stored |
| `status` | no | HTTP status served |
| `ts` | no | Unix ms, default now; up to 7 days back |
| `id` | no | 8–64 char idempotency id: a retried id is ignored |

At most 100 hits and 128 KB per request. Unknown agents and assets are ignored,
not rejected. Response: `{ accepted, ignored, verified, unverified, unknown }`.

### Secrets

- `PVUV_SITE=YOUR_SITE_ID`
- `PVUV_INGEST_TOKEN` — an **ingest-only** token: it can report crawler hits and
  read nothing. Create it with the MCP tool `create_ingest_token` or in the
  dashboard (API tokens → purpose "Crawler sync"). Store it as a server-side
  environment variable / secret. Never expose it to the browser (no
  `NEXT_PUBLIC_` / `VITE_` prefix), never commit it.

## Code by platform

### Next.js (Vercel, any plan, or self-hosted)

Next.js 16+: `proxy.ts` exporting `proxy`; Next.js 12–15: `middleware.ts` exporting
`middleware`. Place it at the project root (or in `src/`). If the project already
has one, add the reporting block to it instead of creating a second file.

```ts
import { NextResponse, type NextFetchEvent, type NextRequest } from 'next/server';

const CRAWLERS = /chatgpt-user|oai-searchbot|gptbot|perplexity-user|perplexitybot|claude-user|claude-searchbot|claudebot|bingbot|googlebot|applebot|duckduckbot|duckassistbot|yandexbot|meta-externalagent|bytespider/i;
const ASSET = /\.(?!(?:html?|txt|xml|md)$)[a-z0-9]{1,8}$/i;

export function middleware(req: NextRequest, event: NextFetchEvent) { // Next 16+: export function proxy(…)
  const ua = req.headers.get('user-agent') ?? '';
  const path = req.nextUrl.pathname;
  if ((req.method === 'GET' || req.method === 'HEAD') && CRAWLERS.test(ua)
      && !path.startsWith('/api/') && !path.startsWith('/_') && !ASSET.test(path)
      && process.env.PVUV_SITE && process.env.PVUV_INGEST_TOKEN) {
    const ip = req.headers.get('x-real-ip') ?? req.headers.get('x-forwarded-for')?.split(',')[0]?.trim();
    event.waitUntil(
      fetch(`https://api.pvuv.ai/v1/sites/${process.env.PVUV_SITE}/crawler_hits`, {
        method: 'POST',
        headers: { authorization: `Bearer ${process.env.PVUV_INGEST_TOKEN}`, 'content-type': 'application/json' },
        body: JSON.stringify({ hits: [{ ua, path, ip, ts: Date.now() }] }),
        signal: AbortSignal.timeout(3000),
      }).catch(() => {}),
    );
  }
  return NextResponse.next();
}

export const config = { matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'] };
```

The middleware runs before the page, so `status` is unknown here: leave it out.
On Vercel set both variables under Project → Settings → Environment Variables
(Production), then redeploy.

### Cloudflare Workers (a Worker that serves or proxies the site)

```ts
const CRAWLERS = /chatgpt-user|oai-searchbot|gptbot|perplexity-user|perplexitybot|claude-user|claude-searchbot|claudebot|bingbot|googlebot|applebot|duckduckbot|duckassistbot|yandexbot|meta-externalagent|bytespider/i;
const ASSET = /\.(?!(?:html?|txt|xml|md)$)[a-z0-9]{1,8}$/i;

export default {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    const response = await handle(request, env, ctx); // the site's existing logic, or fetch(request) for a pass-through
    const ua = request.headers.get('user-agent') ?? '';
    const path = new URL(request.url).pathname;
    if ((request.method === 'GET' || request.method === 'HEAD') && CRAWLERS.test(ua)
        && !path.startsWith('/api/') && !path.startsWith('/_') && !ASSET.test(path)) {
      ctx.waitUntil(fetch(`https://api.pvuv.ai/v1/sites/${env.PVUV_SITE}/crawler_hits`, {
        method: 'POST',
        headers: { authorization: `Bearer ${env.PVUV_INGEST_TOKEN}`, 'content-type': 'application/json' },
        body: JSON.stringify({ hits: [{ ua, path, ip: request.headers.get('cf-connecting-ip'), status: response.status, ts: Date.now() }] }),
      }).catch(() => {}));
    }
    return response;
  },
};
```

`npx wrangler secret put PVUV_INGEST_TOKEN`; put `PVUV_SITE` in `[vars]`.
Cloudflare Pages: the same block in `functions/_middleware.ts`
(`export const onRequest = async (ctx) => { const response = await ctx.next(); …; ctx.waitUntil(…); return response; }`, secrets in `ctx.env`).

### Node.js (Express, Fastify, Koa, NestJS…)

```js
const CRAWLERS = /chatgpt-user|oai-searchbot|gptbot|perplexity-user|perplexitybot|claude-user|claude-searchbot|claudebot|bingbot|googlebot|applebot|duckduckbot|duckassistbot|yandexbot|meta-externalagent|bytespider/i;
const ASSET = /\.(?!(?:html?|txt|xml|md)$)[a-z0-9]{1,8}$/i;

app.use((req, res, next) => {
  res.on('finish', () => {
    const ua = req.get('user-agent') || '';
    if (!['GET', 'HEAD'].includes(req.method) || !CRAWLERS.test(ua)
        || req.path.startsWith('/api/') || req.path.startsWith('/_') || ASSET.test(req.path)) return;
    fetch(`https://api.pvuv.ai/v1/sites/${process.env.PVUV_SITE}/crawler_hits`, {
      method: 'POST',
      headers: { authorization: `Bearer ${process.env.PVUV_INGEST_TOKEN}`, 'content-type': 'application/json' },
      body: JSON.stringify({ hits: [{ ua, path: req.path, ip: req.ip, status: res.statusCode, ts: Date.now() }] }),
      signal: AbortSignal.timeout(3000),
    }).catch(() => {});
  });
  next();
});
```

Behind a proxy or load balancer set `app.set('trust proxy', true)` (or read the
real client address header your host documents) so `req.ip` is the crawler's address.

### Anything else (PHP, Python, Go, Ruby, log files)

Same rule, same POST. For high traffic or log-based reporting, buffer hits and
send up to 100 per request. Never block or fail the page response on it.

## Verify

```bash
curl -s -o /dev/null -A "Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)" https://YOUR-DOMAIN/
```

Then call the MCP tool `get_crawler_hits` with `trust: "all"` (or open the
dashboard's AI & crawlers block): the hit appears as **unverified**, because the
request came from your machine, not OpenAI. Real crawler visits show as verified.