Publishers Are Starting to Charge the Machines That Read Them

For twenty years, the basic online publishing bargain was to rank on Google, receive referral traffic, meaning visitors sent from another site such as Google, and make money from those readers.

In September 2026, Digiday reported that Axios was launching feeds built for AI models and agents, while Search Engine Journal compared three different systems from Google, Cloudflare and Microsoft for paying publishers when AI uses their work.

That shift reaches Answer Engine Optimization, or AEO. The issue is no longer only whether a page can be found, but whether an AI can read it, cite it and create value from it.

What Is Answer Engine Optimization?

Answer Engine Optimization is the practice of making a business, product or piece of writing more likely to be named and cited when an AI system such as ChatGPT, Google's Gemini, Google's AI Overviews, Perplexity or Claude answers a question.

It follows search engine optimization, or SEO, which focused on ranking a web page in a list of links. AEO focuses on becoming a source an AI answer draws on, because the reader may never see that list. Google's AI Overviews are AI-written summaries above its normal search results.

An AI crawler is software an AI company runs to load web pages automatically and copy what is on them, either to train a model or answer a question when it is asked. That makes machine access part of the AEO story as well as citation.

Digiday reported in July that Axios, Forbes, Future, Time and The Washington Post were already packaging AI-visibility data into products for advertisers. Axios chief revenue officer Jacquelyn Cameron told Digiday, "Ensuring that Axios has a high citation score within these LLMs is something we think about a lot."

What Is Axios Direct?

At the Digiday Publishing Summit on Sept. 16 in Key Biscayne, Fla., Cameron said Axios was taking a product called Axios Direct to market.

Axios Direct is built around content feeds, streams of articles delivered by software to another program, the way an RSS feed delivers headlines. Digiday described the product as feeds that "effectively take the concepts of a Bloomberg terminal and an RSS feed and update them for the agentic AI era."

The first feed, launching now, is for investment companies and will contain market-moving reporting, Digiday reported. Companies will pay an annual fee based on their size and assets under management, with two-year contracts.

A second feed, still without a launch date, would let companies pull Axios reporting into internal AI models for employees. Digiday said this differs from AI content marketplaces, where publishers offer material through a third party.

A third feed, which Axios plans to explore in 2027, would serve individual subscribers using personal AI agents. An AI agent is a program that performs tasks on a person's behalf, such as reading the news and summarizing it.

How Are Google, Cloudflare and Microsoft Paying Publishers?

Search Engine Journal reported on Sept. 18 that the three companies are testing sharply different models.

Google's invitation-only AI contribution pilot pays publishers when Google decides their content "contributes significantly" to AI-generated responses in Gemini, AI Overviews and AI Mode. Search Engine Journal reported that Google has not published the criteria or payout math. Google calls the program "an initial learning phase," and Search Engine Journal cited a Digiday report in which one publishing executive said the proposed payments were "too low to opt in."

Cloudflare's Pay Per Crawl charges an AI crawler each time it successfully fetches a page. The publisher sets the price, with a minimum of $0.001 per request per domain, and decides whether each crawler is charged, allowed or blocked. Search Engine Journal said it has been in closed beta since 2025.

Microsoft's Publisher Content Marketplace, announced Feb. 3, lets publishers license premium content for Copilot answers and set their own licensing and usage terms. Microsoft describes payment as based on "delivered value," but Search Engine Journal reported that the company has not publicly explained what triggers payment or how the rate is set.

Search Engine Journal summarized the difference this way: Google judges contribution after the fact, Cloudflare uses a mechanical per-crawl price set by the publisher, and Microsoft has not specified a public trigger or rate.

Who Decides Which AI Crawlers Can Read a Site?

Cloudflare says its network sits in front of more than 20 percent of web domains. On July 1, Cloudflare began sorting AI bots into Search, Agent and Training categories.

Starting Sept. 15, Cloudflare said, Training and Agent bots are blocked by default on ad-supported pages for new customers, new sites and all free customers. Search bots remain allowed. TechCrunch reported that "mixed-use" crawlers are blocked by default on those pages.

The preference is partly expressed through robots.txt, a small text file on a website that tells crawlers which pages they may and may not read; it is a request, not a lock.

Cloudflare's Bot Preference Sync writes robots.txt entries to match the Search, Agent and Training settings selected by a site owner. Search Engine Journal reported that the feature is on by default for new customers as of Sept. 15.

Cloudflare wrote: "When your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules."

How Much Machine Traffic Are Publishers Seeing?

Digiday reported TollBit data showing that publisher sites received one AI bot visit for every 50 human visits in the second quarter of 2025. By the fourth quarter, the ratio was one for every 31.

TollBit said AI bot scraping grew 29 percent from the second to the third quarter of 2025 and 20 percent from the third to the fourth, according to Digiday. Arc XP's network saw a 300 percent year-over-year increase in AI bot traffic.

Arc XP, the publishing platform owned by The Washington Post, says it supports more than 2,500 sites. On March 23, it announced that customers could use TollBit to monitor AI bots, classify them, block them or send them to a paywall that charges AI companies.

Digiday reported that nearly 20 percent of the roughly 7,000 publishers in TollBit's network earned money from that paywall, ranging from hundreds of dollars to tens of thousands of dollars a month.

Why Is AI Visibility So Hard to Measure?

Digiday reported that no standard exists for measuring AI visibility. Mark Howard, Time's chief operating officer, said of the measurement vendors, "They all have different methodologies. Their numbers are all different."

Time uses AI bot traffic to its own site as its main proxy measurement, Digiday reported. The outlet ranks in the 98th percentile for AI bot activity among nearly 7,000 publisher sites in TollBit's network.

Digiday also reported that 56.4 percent of news publishers block at least one AI crawler in robots.txt, while AI-driven traffic grew 187 percent in 2025, roughly eight times faster than human traffic.

A crawl, a citation, a human visit from an AI product and a platform's judgment of "significant contribution" are different events. The payment systems reflect that disagreement: Cloudflare can price a crawl, Google judges contribution, Microsoft refers to "delivered value," and Axios watches citation scores.

What Does All of This Mean?

Taken together, Digiday's reporting on Axios and TollBit and Search Engine Journal's comparison of Google, Cloudflare and Microsoft show several publishing models operating at once. Axios is selling reporting directly to buyers. Cloudflare and TollBit can put a toll gate in front of machine access. Google is testing a system in which the platform decides after the fact what a publisher's contribution was worth.

No common unit of payment has emerged. The systems described by Search Engine Journal and Digiday do not settle on a crawl, a citation, a "significant contribution" or "delivered value."

For a business that publishes anything online, the evidence suggests that machine access is becoming a setting rather than a given. It also suggests caution around AEO measurement. A business owner can compare third-party visibility reports, but the measurements it directly controls remain its own logs of machine visits and its own referral traffic from ChatGPT, Perplexity and Gemini.