Agent, or crawler?

“AI agent traffic” has become a category people report without defining. In practice a request arriving at your server is one of four things, and they need different responses.

The four populations

• Search and index crawlers. Declare themselves, respect robots.txt, arrive on a schedule, traverse breadth-first from your sitemap. Well-behaved and easy to identify.
• Model-vendor crawlers. Collect text for training or retrieval corpora. Increasingly declare themselves too, and increasingly honour opt-out signals. These look like search crawlers and mostly are.
• Link prefetchers. Chat clients, messaging apps and previewers fetching a URL because it appeared somewhere. Arrive once, immediately, usually from a cloud range, often with a browser-shaped user agent. This is the population most often misreported as an agent, because the request is causally downstream of a model emitting a link — but nothing decided anything.
• Autonomous agents. Something that received your content as context and acted on it. Rare, and hard to evidence.

Signals that actually separate them

Traversal shape. A crawler walks structure — sitemap order, breadth-first, many URLs per session. A prefetcher hits exactly one URL and never returns. An agent's path reflects the task it was given, which usually means a small number of semantically related pages with no structural relationship.

Timing. Crawlers are periodic. Prefetchers are immediate and singular. Agent traffic is bursty and correlates with nothing you control.

What was requested. A crawler requests documents. A prefetcher requests the one URL it was handed. Something requesting a URL that only ever appeared inside the body text of a page — never in an href, never in a sitemap — did not traverse to it. It had to read the text.

Network origin. Organisation-level attribution separates residential from cloud, and cloud is where automation lives. It does not tell you which kind of automation, and it never tells you who is behind it.

Signals that look decisive and are not

User agent. Trivially spoofable, and frequently just wrong rather than deceptive — plenty of automation ships a stock browser string with no intent to deceive. Useful as a hint, never as attribution.

Absence of JavaScript execution. Suggestive of a non-browser client, but headless browsers execute scripts and many agent runtimes drive a real browser.

A callback firing. This is the one that causes the most damage. A URL embedded in a page and later requested proves a client followed a link. It does not prove a model read an instruction and complied. Whether it does depends entirely on whether the URL was reachable any other way — and if it appeared in an href or a sitemap, it was.

Why the distinction changes your numbers

If prefetchers and crawlers are counted as agents, every rate built on that count is wrong in the same direction, and the error scales with how popular your content is rather than with how much agent activity exists. A widely-shared page will look like it is under heavy agent traffic when it is under heavy preview traffic.

The practical discipline is to report the population you can defend and name the one you cannot. “Requests that followed a link” is defensible from access logs. “Agents that complied with an instruction” requires evidence that the arrival route existed only inside the text, and most instrumentation does not establish that.

![OpenA2A research marker](https://agentpwn.com/api/canary/ingest/render/apwni_d61073a660bc8e4e)