# When AI asks for a clean copy of your page, does your site answer?

> Some AI tools now ask websites for a clean text copy of each page, using the standard header Accept: text/markdown. We checked 50 B2B marketing agencies on November 10, 2026. Of the 47 we could test, only 11 (23%) answered with a clean copy. Citesprout's own site didn't either, until we fixed it the same day.

By Citesprout · November 10, 2026 · Page for people: https://citesprout.com/resources/ai-asks-for-a-clean-copy

## What changed this year

A normal web page is mostly layout code, which costs AI tools tokens and buries the words. Some AI tools now send the header `Accept: text/markdown`, meaning "send the plain text version of this page if you have one."

- February 2026: Cloudflare launched Markdown for Agents, which answers that header automatically on Pro plans and above. https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/
- February 2026: Checkly tested seven AI coding agents. Claude Code, Cursor and OpenCode asked for the clean copy. OpenAI Codex, Gemini CLI, GitHub Copilot and Windsurf did not. https://www.checklyhq.com/blog/state-of-ai-agent-content-negotation/

## What we found: 50 agencies

- 11 of 47 homepages answered with a clean copy when asked (23%). 9 answered the header directly; 2 offered a copy at /index.md instead.
- 14 agencies passed every basic check but had no clean copy.
- 24 of 47 had no llms.txt file (51%).
- 46 of 47 showed their text without JavaScript. 34 of 47 had structured data.
- 3 of 50 asked major AI crawlers to stay away in robots.txt.

## Two ways to serve AI, and why one is safer

1. Matching bot names (ClaudeBot, GPTBot) misses AI agents that act for a person and look like normal visitors.
2. Google's John Mueller has criticized separate bot-only pages, partly over cloaking worries (Search Engine Journal, February 13, 2026: https://www.searchenginejournal.com/cloudflares-new-markdown-for-ai-bots-what-you-need-to-know/567339/). Google hasn't said how it views the header approach.

Answering the standard header is cleaner: anyone who asks gets the same words in a simpler format, and browsers never send the header. Until November 10, 2026, Citesprout only answered named bots; it now answers both ways.

## Test your own site

```
curl -s -o /dev/null -w "%{content_type}\n" -H "Accept: text/markdown" https://yoursite.com/
```

text/markdown means your site answers. text/html means it doesn't. Free check without a terminal: https://citesprout.com/ai-view

## How to fix it

- Cloudflare Pro or above: turn on Markdown for Agents, check the output, and review its default AI-use signals.
- Any other host: prepare a clean copy of each key page, add a server rule that sends it when the header is present, and send Vary: Accept.
- Keep the content the same as the page.

Citesprout sets this up in a Crawlability Fix: https://citesprout.com/services

## Honor roll: 8 agencies that passed every check

Lean Labs, Virayo, Ten Speed, Understory, Skydog Ops, RevenueHoop, WarmUp, Kula Partners.

## How we checked

November 10, 2026. Homepages of 50 B2B marketing agencies from our own list. Our research bot identified itself, read only public pages (homepage, robots.txt, llms.txt, possible clean copy), paused between sites, skipped sites whose robots.txt asked bots not to crawl, and did not pretend to be any AI company's bot. Three sites couldn't be tested. A one-day snapshot of the basics, homepages only. Re-checks and corrections: https://citesprout.com/contact

## Questions

### How do AI tools ask a website for a clean copy of a page?

They send the HTTP header Accept: text/markdown. A site that supports it answers with a plain text version of the same page. Browsers never send this header.

### Which AI tools ask for a clean copy?

In Checkly's February 2026 test: Claude Code, Cursor and OpenCode. Not OpenAI Codex, Gemini CLI, GitHub Copilot or Windsurf. This changes as tools update.

### Is serving AI a different version of a page cloaking?

Google has not said. The safest approach keeps the content the same, changes only the format, and answers anyone who asks through the standard header.

### How do I add this to my site?

Cloudflare Pro or above: turn on Markdown for Agents. Otherwise: a server rule that answers the header with a prepared text copy of each page, which Citesprout sets up in a Crawlability Fix.