---
title: "Markdown for Agents on Free Tier Cloudflare Workers · Blog"
description: "How to replicate Cloudflare's paid Markdown for Agents feature with a free-tier Worker: content negotiation, regex-based HTML to Markdown, and a 75% token reduction for LLM crawlers."
source: "https://adhenawer.net/blog/en/markdown-for-agents-free-tier.html"
lang: en
---

Cloudflare ships a feature called *Markdown for Agents* that returns a clean Markdown version of any page when a client sends `Accept: text/markdown`. It is gated behind Pro and Business plans. For a static blog hosted on GitHub Pages, paying for an edge plan just to help crawlers parse articles is hard to justify. This post walks through a 170-line Cloudflare Worker that replicates the feature on the free tier for `adhenawer.net`, and explains the trade-offs.

## Why bother serving Markdown at all

Each article on this site is a self-contained HTML file with theme switcher, progress bar, resume banner, JSON-LD, Open Graph tags, and inline navigation. A single article weighs around 53,000 tokens when an LLM ingests the raw HTML. Most of that is markup the model will discard. Serving the same content as Markdown with a YAML frontmatter brings the same article down to roughly 13,000 tokens. That is a 75% reduction, which matters for any retrieval pipeline that pays per token or has a finite context window.

The trick is content negotiation. A browser asks for `text/html` and gets the full HTML. An agent asks for `text/markdown` and gets Markdown. The URL stays the same. No separate endpoints, no sitemap duplication, no new paths to index.

## The Worker in one read

The entry point checks the `Accept` header. If it does not include `text/markdown`, the Worker simply proxies the request to the origin and returns whatever GitHub Pages served. No transformation, no overhead beyond a `fetch`.

```js
function wantsMarkdown(request) {
  const accept = request.headers.get("Accept") || "";
  return accept.includes("text/markdown");
}

export default {
  async fetch(request, env, ctx) {
    if (!wantsMarkdown(request)) {
      return fetch(request);
    }
    // ...fetch origin as HTML, transform, return Markdown
  }
};
```

When Markdown is requested, the Worker fetches the same URL from the origin with `Accept: text/html` forced, then extracts the `<article>` block, strips the chrome, and converts what remains.

## Why regex instead of a parser

The obvious way to convert HTML to Markdown is to use Turndown with JSDOM. Neither runs cleanly in the Workers runtime. JSDOM pulls in Node built-ins that are not available, and bundling DOM parsers blows past the 1 MB Worker script limit on the free tier. Cloudflare does offer `HTMLRewriter`, which is fast and streaming, but it is designed for targeted rewrites rather than full tree-to-text conversion.

Since the source HTML is generated by my own `build_html.py`, its structure is predictable. A sequence of regex passes is enough. This is not a general-purpose converter. It is a converter for exactly the HTML this site produces.

```js
// Remove elements we don't want
md = md.replace(/<div class="theme-bar">[\s\S]*?<\/div>/gi, "");
md = md.replace(/<a[^>]*class="back-home"[^>]*>[\s\S]*?<\/a>/gi, "");
md = md.replace(/<script[\s\S]*?<\/script>/gi, "");
md = md.replace(/<style[\s\S]*?<\/style>/gi, "");
md = md.replace(/<nav[\s\S]*?<\/nav>/gi, "");

// Convert headings
md = md.replace(/<h2[^>]*>([\s\S]*?)<\/h2>/gi, "## $1\n\n");

// Convert links
md = md.replace(/<a[^>]*href="([^"]*)"[^>]*>([\s\S]*?)<\/a>/gi, "[$2]($1)");

// Strip whatever tags remain
md = md.replace(/<[^>]+>/g, "");
```

Order matters. Links must be converted before the generic tag stripper, otherwise the `href` is lost. Headings must be converted before paragraphs, otherwise inner text gets wrapped twice. Script and style blocks must be deleted first because they contain characters that would otherwise confuse the regex for anchors and attributes.

## Frontmatter and the token header

The Worker pulls `<title>`, `meta name="description"`, and `meta name="author"` from the original HTML and builds a YAML frontmatter block. It also emits an `x-markdown-tokens` response header with an estimated token count (length divided by four, good enough for budgeting).

```js
return new Response(finalMarkdown, {
  status: 200,
  headers: {
    "Content-Type": "text/markdown; charset=utf-8",
    "x-markdown-tokens": String(tokens),
    "Vary": "Accept",
    "Cache-Control": "public, max-age=3600",
    "Access-Control-Allow-Origin": "*",
  },
});
```

The `Vary: Accept` header is not optional. Without it, any cache in front of the Worker, including Cloudflare's own, may serve the HTML version to an agent or the Markdown version to a browser depending on who asked first. With `Vary: Accept`, the cache keys on the header and both variants coexist.

## Cost and limits

The Workers free tier gives 100,000 requests per day and 10 ms of CPU per request. The regex pipeline on a 50 KB HTML file runs in a couple of milliseconds. A small blog will never come close to either ceiling. Egress is free, and there is no charge for the subrequest to GitHub Pages.

The trade-offs are worth naming. The converter is fragile against HTML it was not designed for. Add a new component to the template without updating the Worker and the Markdown output will contain leaked markup or missing content. There is no test suite shipped with the Worker; the feedback loop is a `curl` in the terminal. And regex cannot handle nested lists or tables well. For a transcription blog where articles are paragraphs, headings, and occasional figures, that is fine. For a documentation site with code samples and tables, a real parser would be worth the extra setup.

## Try it

The full source is at [github.com/adhenawer/video-to-text](https://github.com/adhenawer/video-to-text) under `workers/markdown-agent/`. To see it in action:

```bash
curl -H "Accept: text/markdown" https://adhenawer.net/posts/pt_br/chefe-do-claude-code-o-que-acontece-depois-que-a-programacao-for-resolvida.html
```

Same URL, different `Accept`, a fraction of the tokens. No plan upgrade required.