The Index You Never Licensed

Brave Search sold its index to AI firms. That sentence alone should stop most publishers mid-scroll, because the entire pitch of Brave's search product was that it was not Google, not Bing, and not beholden to the same data arrangements as the big platforms. It crawled the web independently, built its own index, and offered that as a point of pride for anyone who wanted an alternative to the dominant search engines. The reporting on the sale makes clear that sites which were crawled for search results, the ordinary and uncontroversial purpose of a search index, are now part of a dataset licensed to AI companies for training, and a lot of those sites never agreed to that second use at all.

This is not a story about Brave being unusually dishonest. It is a story about what happens when the thing being governed by consent mechanisms turns out to be more valuable, and more transferable, than anyone building those mechanisms accounted for. The index was never just a tool for serving search results. It was an asset. And assets get sold.

Consent was built for one transaction, not two

A robots.txt file, a noindex tag, an AI-training opt-out flag: these are all instructions given in the context of a specific relationship. A site owner lets a crawler in because the crawler returns the favour by sending ranked traffic back. That is the deal search indexing has run on for two decades. It was never written to anticipate that the index itself, once built, might be treated as a standalone asset that could be sold to a party the site owner has never heard of, for a purpose entirely disconnected from search results.

Search Engine Journal's own coverage of AI training opt-outs makes a related point from a different angle: rank data problems and opt-out signals are already inconsistent across tools and crawlers, which means the infrastructure for saying no was shaky even before anyone considered that an entire index might simply be sold wholesale. An opt-out flag that already behaves unpredictably within its original context offers no protection at all once the index it sits inside becomes a resellable product passed to a buyer the site owner never dealt with directly.

It is worth being precise about what broke here, because it is not a single failure but a chain of them. First, the signal itself, an opt-out or a crawl directive, was written with only one downstream use in mind. Second, the enforcement of that signal depended entirely on the goodwill of the party doing the crawling, with no independent check. Third, once the crawl was complete and the index existed as a standalone product, the original signal had nothing left to attach to. It travelled with the crawl, not with the sale.

Indexing as an asset class

The uncomfortable truth is that an index is just data, and data that AI companies want to buy has a price. Once a search engine's crawl becomes something that can be licensed rather than only served, the incentives shift from "build the best index to answer queries" to "build an index that can be monetised twice." Every site that was crawled under the first premise is now, without having agreed to anything new, an ingredient in the second.

This is the part that consent frameworks were never designed to handle. Terms of service, meta tags, and crawl directives all assume a closed loop: crawler comes in, results go out, value is exchanged in the form of traffic. A resale breaks that loop entirely. The site owner's original "yes" to being crawled was never a "yes" to becoming training material for a third party they cannot name and did not choose. There was no clause anyone signed that covered this scenario, because nobody writing those clauses imagined an index as a tradeable good in its own right.

Compare this to how other industries treat resale of licensed material. A photo licensed for one publication cannot simply be relicensed by the publisher to a third party without fresh permission from the photographer. Search indexing has operated for years without an equivalent principle, because the index was treated as infrastructure rather than inventory. The Brave sale demonstrates that the distinction has quietly disappeared, and nobody updated the rules to match.

Consent theatre versus consent in practice

MarTech's recent piece on engineering consent for better marketing argues, reasonably, that opt-in mechanisms should be designed with more care rather than bolted on as an afterthought. That is a sound idea for email lists and ad targeting, where the relationship between the party collecting data and the party using it tends to stay fixed. It falls apart as a model for search indexing the moment the index itself can be sold on, because no amount of careful opt-in design at the point of crawling can anticipate a sale that happens years later, to a buyer the original crawl agreement never mentioned. Engineering consent assumes the terms of the transaction are known in advance. The Brave sale shows that, for indexing, they frequently are not, and cannot be, because the second buyer does not exist yet when the first crawl happens.

The gap is structural, not a matter of better wording on a consent banner. A site owner who permitted crawling for search results has no mechanism to object to, or even learn about, a downstream licensing deal unless a journalist happens to report on it. That is not consent. That is a hope that nobody notices before the deal closes.

The measurement problem compounds it

There is a second layer worth naming: even the tools built to measure how visible a site is inside AI systems are, per Search Engine Journal's own analysis, unevenly reliable. If visibility inside AI tools is already hard to measure accurately, then sites swept into a resold index have no clean way to even establish what their content is now being used for, let alone whether they are being represented accurately within it. Consent failed at the point of collection, and measurement fails at the point of oversight. Neither failure helps the other, and a site owner trying to work out what happened to their content is left with two broken instruments instead of one.

What this actually settles

The lesson is not that Brave is the villain of search. It is that the entire architecture of robots.txt, noindex, and AI-training opt-out flags was built on an assumption that is now false: that an index, once built, stays inside the relationship it was built for. It doesn't. It can be sold, licensed, bundled, and resold again, and every signal a site owner ever sent about how their content should be used was sent into a system that had no obligation to honour it once ownership of the index changed hands.

Fixing that is not a job for better consent copy or friendlier opt-in prompts. It requires treating the resale of an index as a distinct act requiring its own permission, separate from the permission granted for the original crawl. Until that distinction exists in practice rather than only in argument, every site that has ever allowed itself to be indexed by anyone is, in effect, licensing its content for purposes nobody has told it about yet.