What happened
Cloudflare reported on 4 August 2025 that it had observed Perplexity crawling sites that had blocked its declared crawler. When the declared crawler was blocked, Cloudflare reported seeing a generic browser intended to impersonate Google Chrome on macOS, making requests from addresses outside Perplexity’s official ranges, with the user agent and source networks repeatedly changed to hide the activity. The test used brand new domains that, in Cloudflare’s words, had not yet been indexed by any search engine nor made publicly accessible in any discoverable way. Cloudflare stated that it had de-listed Perplexity as a verified bot and added heuristics to its managed rules to block the crawling.
Perplexity published a reply the same day, dismissing the research as a publicity exercise and a traffic analysis failure. Its substantive answer was one of misattribution: it stated that Cloudflare had misattributed three to six million daily requests from BrowserBase’s automated browser service to Perplexity. That dispute is not resolved in the sources reviewed for this entry.
The original Wired report could not be retrieved for this entry, and the account that follows rests on outlets reproducing the Wired text. It was reported by Wired in June 2024 that a machine hosted on Amazon infrastructure, and not on Perplexity’s published crawler list, had hit properties belonging to Condé Nast, the publisher of Wired, at least 822 times in three months despite robots.txt blocks. It was reported that Wired built a test page containing a single sentence, and that the product returned an invented story about a girl following glowing mushrooms in a forest, while Wired’s server logs recorded that it never attempted to visit the page. It was also reported that the product told users Wired had reported that a California police officer had stolen bicycles from a garage, and that Wired had published no such report.
What an auditable version would have shown
It cannot be established from outside which sources any particular answer was built from. An auditable version answers that from the agent’s side. A record written for each retrieval says what was requested, from which address and under what declared identity, and whether the request honoured or bypassed an exclusion. A record written for each answer says which retrieved documents were before the model when the response was generated. With those two records, whether an answer rests on a source it names can be checked by reading a log rather than by a publisher testing for it. The identity a crawler presents and the provenance of an answer can instead be checked from records rather than inferred from traffic.
Where the gap was
Cloudflare’s research states that requests continued under a browser identity after the declared crawler was blocked, and Perplexity disputes that the traffic was its own. A ConstraintGate is designed to test a publisher’s exclusion before a fetch is made. A request to an excluded host is refused, and any authorised override is recorded as an explicit decision. The identity a request carries is then a matter of record on the agent’s side. It was reported that the product produced text attributed to a publication that had published nothing of the kind. A CitationVerifier tests each citation against the source it names before an answer is delivered to a user, so an attribution that no retrieved document supports is caught at the point of generation. A ConductRecord preserves the request, the declared identity, the documents retrieved and the output. That allows a disputed answer to be traced afterwards rather than reproduced by a party trying to trigger it again. None of these controls resolves who was responsible for the traffic Cloudflare measured. They mean the record needed to answer that question exists on the agent’s side.
What governance should have looked like
Cloudflare’s findings are contested by Perplexity, whose position is that a substantial share of the traffic belonged to a third party service, and this entry does not resolve that. The 2024 findings rest on Wired’s reporting as carried by other outlets rather than on the original, which could not be retrieved. Where an organisation operates a retrieval agent across the open web, best practice would be for every outbound request to carry a declared and stable identity. Publisher exclusions would be enforced as a check before the request rather than as a policy statement afterwards, and third party retrieval performed on the organisation’s behalf would be identified as such. Each answer would also carry a record of the documents it was actually built from, so an attribution can be tested against them. Several of the copyright claims listed in the sources below are live and untested, and this entry will need revisiting as they are decided.
Failure Pattern: a network operator said undeclared crawler traffic reached material a publisher had excluded, which the company disputed, and a publication separately reported that the product produced text attributed to it that it says it did not publish, with no agent side record tying a given output to the sources actually retrieved.
Governance Principle: an organisation operating a retrieval agent should be able to show, for any answer it produced, which sources were actually fetched and under what identity the request was made, and to establish that a source it names supports the claim attributed to it.
The reference implementation of ConstraintGate, CitationVerifier and ConductRecord is open source. It lives at github.com/saffronandindia/headlights-oss, Apache 2.0 licensed and free to install. The repository is public now.
Sources
- Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (Cloudflare, 4 August 2025)
- Agents or bots? Making sense of AI on the open web (Perplexity AI, 4 August 2025)
- Perplexity AI CEO Aravind Srinivas on plagiarism accusations (Fast Company, 21 June 2024)
- Dow Jones & Company, Inc. v Perplexity AI, Inc., opinion and order of 21 August 2025 (S.D.N.Y., 1:24-cv-07984, via Justia)
- Britannica files copyright and trademark infringement lawsuit against Perplexity (Encyclopaedia Britannica, September 2025)
- Japan newspapers sue AI startup Perplexity (The Japan Times, 26 August 2025)
- The New York Times is suing Perplexity for copyright infringement (TechCrunch, 5 December 2025)
- Wired confirms Perplexity is bypassing efforts by websites to block its web crawler (MacStories, June 2024, reproducing the Wired finding on the 822 requests)
- Report on the Wired test page and the invented story returned for it (Futurism, 22 June 2024)
- Republication of the Wired reporting, carrying the 822 requests figure and the fabricated police report attributed to Wired (IT Magazine, 20 June 2024)