1txtfirst

4.3 KB as /index.md, updated 2026-09-07, sha256 bd94f3c64155

You are reading a rendering. The page is the markdown file behind it.

Every URL on this site answers in two shapes from a single source. Ask for it as a human and you get minimal HTML. Ask for it as a machine and you get the file itself.

curl https://txtfirst.com/spec                      # HTML for a browser
curl -H "Accept: text/markdown" https://txtfirst.com/spec   # the source
curl https://txtfirst.com/spec.md                   # the source, no headers needed

No build step produced two versions of this text. There is one file. The HTML is generated from it at build time and thrown away on the next build.

Check your own site first

curl -s https://txtfirst.com/cli/index.js > txtfirst.js
node txtfirst.js check yourdomain.com

Nothing to install beyond Node, no account, nothing sent anywhere. The checker is one file with no dependencies, served from this site, and you can read it before you run it. It reports whether your origin answers a plainly identified program at all, and then whether it meets the four requirements on spec. The same check runs from the form at the top of this page if you would rather not open a terminal.

Most sites fail the first question before reaching the second, which is the finding this whole site exists to report.

The claim

The web was written for a reader that renders. It is now read, at scale, by a reader that parses. Those two readers want opposite things, and for thirty years only one of them got a format designed on purpose.

txtfirst says: write the document once, in the format the parser wants, and render it down for the human. Not the other way around. Markdown first, HTML second.

Why this is not just a nicer llms.txt

llms.txt is a map you maintain by hand next to the real site. It drifts, because nothing breaks when it does. txtfirst has no second copy to drift: the file the agent reads is the file the page is made of. If the page is wrong, the markdown is wrong, and you fix it in one place.

What it is worth, measured

On 4 September 2026 we asked twenty five hotel and travel homepages for their front page, using an honestly identified HTTP client. Eight answered with readable content. Seventeen returned 403, 429, a timeout, or a page with no text in it.

Across the eight that answered, the median page was 641 KB of HTML, the median text payload was 7.7 KB, and the median ratio between them was 78 to 1. Those are three separate medians and no single site holds all three. One site, riu.com, sent 5,822 KB to deliver 0.8 KB.

The first problem is not that machine readers are served badly. It is that most of the web does not answer them at all.

Method, per-site results and the raw JSON are on measure. The numbers on this page come from that run and nothing else.

What it costs you to adopt

One middleware function and a content directory. The reference implementation on implement is forty five lines of JavaScript, and the same logic is shown for Cloudflare Pages, Vercel, nginx and Apache. You keep your existing HTML site if you want. txtfirst is additive.

Who runs this

Three publishing sites serve their article pages this way in production, from one shared patch, and each passes fourteen live checks. Their home pages do not, and score three of eight, which is what the checker will tell you if you point it at them. That gap is real and it is left visible here rather than described as full adoption.

The verification samples one article per site, not every article. The patch itself is published as a script rather than as instructions, at /adopt/, because a specification with one implementation is a blog post.

What this site itself is asked for, by shape and by client, is generated from its own logs every day at reads. It is small and young and says so.

Start here

• spec, the four requirements, version 0.1
• why, the argument, including where it does not hold
• measure, the September 2026 readability study
• implement, copy and paste it

txtfirst is a proposal, not a product. Everything here is CC0. Take it, fork it, rename it, ship it without credit.

2txtfirst 0.1

3.6 KB as /spec.md, updated 2026-09-04, sha256 92b4fbaa91c3

A site is txtfirst compliant when it meets all four requirements below. Compliance is per origin, not per page. The key words MUST, SHOULD and MAY follow RFC 2119.

R1. One source, in markdown

Each addressable page MUST be generated from exactly one markdown file. That file is the canonical document. The HTML MUST be derivable from it. If a fact appears in the HTML that is not in the markdown, the site is not compliant.

This is the whole spec in one line. The other three requirements only say how to hand that file to a client.

R2. Content negotiation

For a page at /path, the origin MUST serve the markdown source when either condition holds:

1. the request carries Accept: text/markdown, or
2. the request path is /path.md

The response MUST use Content-Type: text/markdown; charset=utf-8. The two forms MUST return byte-identical bodies.

The HTML response MUST advertise the source with both a link header and a link element:

Link: </path.md>; rel="alternate"; type="text/markdown"
<link rel="alternate" type="text/markdown" href="/path.md">

Requirement 2 exists because most clients cannot set headers. Requirement 1 exists because the URL suffix is a hack and the header is the correct mechanism. Ship both.

R3. Discovery

The origin MUST serve /.well-known/txtfirst.json listing every page, its HTML URL, its markdown URL and its size in bytes. The origin SHOULD also serve /llms.txt and /llms-full.txt for clients that already look for them.

{
  "txtfirst": "0.1",
  "negotiation": { "header": "Accept: text/markdown", "suffix": ".md" },
  "index": "https://example.com/llms.txt",
  "pages": [
    { "url": "https://example.com/rates",
      "markdown": "https://example.com/rates.md",
      "title": "Rates", "bytes": 2214 }
  ]
}

robots.txt MUST NOT block the markdown paths, and SHOULD carry explicit Content Signals stating whether the content may be used for search, for AI input and for AI training. Silence is not consent and it is not refusal either. State it.

R4. Figures that do not require vision

A compliant page MUST NOT put information in a raster image that appears nowhere in the markdown. In practice:

• Diagrams SHOULD be inline SVG with real <text> elements, or ASCII inside a fenced code block. Both are text to a parser and both survive a copy and paste.
• Charts MUST be preceded or followed by the table they were drawn from.
• alt MUST state the fact, not describe the picture. alt="Q3 occupancy 71 percent, up 4 points from Q2" and not alt="bar chart".
• Photographs MAY be raster. A photograph carries no facts, so nothing is lost.

R4 is the requirement people skip. It is also the one that makes the other three worth having, because a page whose numbers live inside a PNG is unreadable no matter how cleanly you serve it.

Conformance

An origin claiming txtfirst 0.1 SHOULD publish the result of a self test at /.well-known/txtfirst.json. There is no certifying body and there will not be one.

Non-goals

txtfirst does not define a markdown flavour, does not require you to drop your existing site, does not replace schema.org, and does not care what your HTML looks like. It defines where the truth lives and how to ask for it.

Changes

• 0.1, 4 September 2026. First public draft. Open questions: whether R2 should require Vary: Accept, and whether a compliant origin must serve markdown for paginated or personalised pages at all.

3Fact Block 0.1

6.1 KB as /dialect.md, updated 2026-09-04, sha256 7fe5505aa908

A Fact Block is eight named fields, always the same, always in the same order, under a markdown heading. It invents no syntax. That is the entire idea.

## Learn Salsa In Paradise, San Juan
answer: from 30 USD, rated 5 out of 5 from 156 reviews, 60 min
where: San Juan, Puerto Rico, United States
price: from 30 USD
duration: 60 min
rating: 5/5 from 156 reviews
booking: instant confirmation, free cancellation
source: https://www.viator.com/tours/San-Juan/...d903-288497P1
checked: 2026-09-04

Do not invent a language

The obvious move, once you decide to write for machines, is to design a compact notation of your own. It is the wrong move and it is worth saying why, because the reasoning is not obvious.

A language model is fast at what it saw a trillion times during training. A notation you invented last week is out of distribution: the model has to spend reasoning on decoding it before it can use it, and it will not tell you when it decodes it wrong. Tokenisers are trained on real text, so invented sigils and clever abbreviations often cost more tokens than the plain English they replace. You would be paying a tax and calling it a saving.

So the language is the one the model already knows. What you get to design is the shape. A Fact Block is fast to read because it never varies, not because it is novel.

The eight fields

All eight, in that order, or it is not a Fact Block. Omitting a field you do not have is not allowed, because a reader cannot tell an absent field from a forgotten one. The value becomes not stated and stays visible.

A gap is allowed in a field and not allowed in answer. A line that reads *"from not stated USD"* is not quotable, so the gate refuses the whole block rather than publish a hollow summary.

The answer line is computed, never written

answer is assembled from the fields below it by string formatting. No language model writes it. A model asked to summarise a record with a gap in it will fill the gap, confidently, and nothing downstream will notice. Composing the line arithmetically removes the opportunity rather than checking for it afterwards.

The gate

A block that cannot be trusted does not get published. The reference implementation refuses to emit one when:

• source is missing or is not a URL
• checked is missing, malformed, in the future, or more than 180 days old
• fields are out of order, or any field is absent
• answer carries fewer than two numbers, which means it is a sentence and not an answer
• answer contains a gap, so it is not quotable alone
• answer cites a number that appears in no value field below it

That last rule is the one that earns its place, and it took two attempts to write. The first version compared the answer against every other line, source and checked included. A Viator URL ends in d903-288497P1 and a date contains 2026, so an answer reading *"from 903 USD, rated 2026 out of 5 from 288,497 reviews"* passed, with all three numbers invented and all three corroborated by digits in a URL slug. The rule now compares against value fields only.

It is worth saying plainly what that means: the check that a summary matches its data was, for a few hours, satisfiable with noise. It was found by a review, not by its own tests.

Why a fixed shape helps a machine

• The model copies rather than interprets, and a model that copies cites.
• It is extractable with one regular expression, so a consumer needs no parser.
• It is auditable. A shape you can check is a shape you can refuse.

It is not mainly a compression win, and an earlier draft of this page said it was. The example block above is 361 bytes. The same nine facts written as natural prose, carrying the same URL, are 490 bytes. That is 1.36 times, not the three times this page claimed before the number was measured. Strip the URL from both and it is 1.46. Real, small, and not the reason to do this. The reason is the fixed shape.

Why it helps a person too

The block is readable out loud. Nobody needs a legend, because the field names are the legend. This matters more than it sounds: a format only a program can read gets no proofreading, and unproofread data goes stale silently.

Reference implementation

factblock.py, 231 lines, no dependencies, CC0. The file asserts its own length, because this sentence would otherwise rot on the next edit. It generates blocks from the Viator Partner API and gates them.

python3 factblock.py --site path/to/config.json   # generate
python3 factblock.py --check                      # self-test the gate

The generator is written against one source because that is the one we could test against real records. The gate is source agnostic and is the part worth reusing.

What this does not solve

A Fact Block makes a record cheap to read and hard to falsify by accident. Three limits, all of them real:

• The gate checks shape, not truth. It cannot tell whether 30 USD was ever the price. It only checks that the same 30 appears in every place that claims it.
• 180 days is a guess. It is far too long for a price and probably fine for a duration, and the format has one window for both.
• A well shaped block full of stale prices is worse than prose, because it looks checked.

Relationship to the spec

The spec says how a document is served. This says what one can look like inside. R1 to R4 stand on their own and a site can meet all four without ever writing a Fact Block. This is a profile, not a fifth requirement.

4Why

4.8 KB as /why.md, updated 2026-09-06, sha256 fea38d4ff6cc

The short version

A document written for a parser and rendered for a person is cheaper to serve, cheaper to read, has nothing to fall out of sync with, and is no worse for people. A document written for a browser and scraped by a parser is none of those things.

Three arguments that hold

1. The reader you are optimising for changed

Page weight has been justified for twenty years by what it buys the human: layout, interactivity, brand. That trade is real and it is fine. What is new is that a large and growing share of your readers discard all of it. They fetch, strip and parse. For those readers the entire investment is overhead, and there is currently no way for them to ask you for the cheap version. txtfirst is that way.

2. Two representations, one source, and drift you can see

Every organisation that has tried to serve machines has done it by maintaining a second artifact: an API, a feed, an llms.txt, a knowledge base. Second artifacts drift, because the site breaking is loud and the feed going stale is silent. Deriving both outputs from one file makes the failure loud instead of silent, which is a smaller claim than removing it and the only one worth making.

This site broke that rule on its first day. The build produced the combined llms-full.txt by re-reading the source directory instead of reusing the bytes it had already served, and that second read path skipped a substitution the first one applied. The file for machines was the one file that went out wrong. A second read path is a second copy. The build now asserts that no artifact leaves it with an unresolved placeholder.

3. The cost is asymmetric and lands on the reader

Serving 641 KB to deliver 7.7 KB of text costs you a rounding error in bandwidth. It costs the agent reading you a real amount of context and money, and context is the scarcest thing an agent has. Whoever is cheapest to read gets read more often and gets read further. That is not a moral argument, it is a budget.

Two arguments that do not hold

"Plain text ranks better"

It does not. Google indexes HTML, and a page served as bare text with no semantic markup, no headings and no internal links will rank worse than the same content in ordinary HTML. If you strip the HTML you lose classic search, and classic search is still where most AI citations are sourced from.

This is why R1 says markdown is the source and not that HTML is forbidden. A txtfirst site has full semantic HTML, proper metadata and schema.org. It just generates them from the file instead of hand editing them.

"Agents prefer markdown, so this wins immediately"

Most crawling today still requests HTML and never sets an Accept header. Adopting txtfirst does not redirect existing traffic to a cheaper path on day one. What it does is make the cheap path exist, and cost you almost nothing to maintain until clients start asking for it. Treat it as an option you are buying, not a switch you are flipping.

The honest summary

The gain today is a smaller surface, a single source, and a site that is trivially legible to anything that parses. The gain later depends on clients that mostly do not exist yet. Adopt it because it is a better way to build a content site, and let the agent readability be the part that ages well.

Where the idea comes from

It is not new. README files, RFCs as plain text, Jeremy Howard's llms.txt, Cloudflare serving markdown variants at the edge, and every static site generator ever written all point at the same thing.

Closest in practice is context.dev, a commercial web scraping API, and it is worth being precise because a first draft of this paragraph was not. Measured on 6 September 2026: every marketing page negotiates on Accept: text/markdown, offers a .md suffix, and a sitemap.md documents both. That is R2, in production, on a funded product. On top of that, the homepage alone also hands markdown to any client whose user agent does not look like a browser, whatever Accept says. So the client can choose everywhere, and on one page the server also guesses.

Two things it does not do. Nothing tells a program where a page's own source lives: no Link header, and the one alternate element on a page points at /auth.md, not at that page's markdown. And there is no way to check that the markdown and the HTML came from one file. And the markdown surface is uneven: the blog index answers 366 bytes of markdown against 584 KB of HTML, which is a stub next to a page, not a rendering of it. R1 is the difference. Serving markdown is the easy half.

txtfirst only insists on the direction of the arrow: markdown is not an export of the page, it is the page.

5How readable is hospitality, actually

8.0 KB as /measure.md, updated 2026-09-04, sha256 f74e56f3b140

Answer: eight of twenty five sites returned readable content to a plainly identified HTTP client. The other seventeen returned an error, a timeout, or a page with no text in it. Among the eight that answered, the median HTML weight was 641 KB, the median text payload was 7.7 KB, and the median ratio between them was 78 to 1.

Those three medians come from three separately sorted columns. No single page in the sample holds all three values, and quoting them as one page would be wrong.

Measured 4 September 2026. Raw results: probe-2026-09-04.json.

Method

A single GET to each origin's homepage from one datacenter IP, no JavaScript execution, no cookies, no retries, 25 second timeout. Two user agent strings were tried against every site:

• browser, a current Chrome string
• agent, TxtFirstProbe/0.1 (+https://txtfirst.com/probe; research on agent readability), honest and contactable. That URL resolves: see probe for what it requests and how to block it.

Text was extracted by removing script, style, noscript, svg, template and iframe elements, then all remaining tags, then collapsing whitespace. A result counts as no content when the request failed or when fewer than 1,500 bytes of text survived extraction. The threshold is arbitrary and is stated so you can disagree with it.

Results, agent user agent

What stands out

Identifying yourself honestly changed almost nothing, and what it changed looks like noise. The agent string got through on 8 of 25, the Chrome string on 7. Those totals hide the more interesting detail: three sites flipped. The agent string won airbnb.com and tripadvisor.com and lost trivago.com, which answered Chrome with a 200 and the probe with a 403. With one request per cell and no repeats, a difference of one site is not a finding. What can be said is narrower and still worth saying: there is currently no mechanism by which a well behaved reader can be treated better than a badly behaved one, and nothing in this sample suggests one exists.

A success code is not an answer. agoda.com and riu.com returned 200 and booking.com returned 202, and none of the three carried readable text. A status code does not measure whether you were understood. riu.com sent 5,822 KB to deliver 0.8 KB of text.

The ratio is worse than the raw weight suggests. The leanest site in the sample, wyndhamhotels.com, still spends 21.5 bytes for every byte of text. The heaviest that still answers, mandarinoriental.com, spends 197.3.

The 403 on our own robots.txt

From the moment its Cloudflare zone was created until it was found and fixed on 4 September 2026, txtfirst.com returned HTTP 403 to a client identifying as Python-urllib/3.14, on /, on /spec.md and on /robots.txt. That last one is the part worth keeping: the site denied a machine the one file that exists to tell machines what they are allowed to do. Nobody configured that. It is Cloudflare's Browser Integrity Check, which is on by default on a new zone. It was never switched on, it was never off.

Measured on the same day across thirteen unrelated domains on separate zones created at different times and untouched for this, the block is narrow: Python-urllib gets 403 while curl, python-requests, Googlebot, GPTBot and ClaudeBot all get 200. So the honest version of this is not that the web refuses machines. It is smaller and, for anyone writing a client, worse:

A site can refuse the Python standard library's default HTTP client without anyone deciding to, and you cannot see it from a browser. urllib is what urllib.robotparser uses to fetch robots.txt, so a well behaved script that checks permission before crawling is the one that gets turned away, while the same script with a made up user agent string sails through. Politeness was the failure mode.

One consequence for the table above: some share of the seventeen non-answers may be an inherited default rather than a decision. This study cannot tell those apart, and the sentence in the limitations below that attributes them to datacenter IP mitigation is one guess among at least two. Separating them takes the same sweep run with a curl user agent from a residential address, which has not been done. And whatever you believe your machine readability policy is, it is worth checking with the client you would actually write:

curl -A "Python-urllib/3.14" -o /dev/null -w "%{http_code}" https://yoursite/robots.txt

Browser Integrity Check has been turned off here, and the conformance script now checks three plain user agent strings on every run.

Limitations, stated plainly

The sample is small and every cell is a single observation. Twenty five sites, one request per site per user agent, no repeats. Eight readable results is not enough to support a claim about a difference of one or two sites, which is why the section above walks its own conclusion back. The medians are medians of eight numbers and should be read as an order of magnitude, not a benchmark.

It is also one IP without a browser engine. A real agent driving a headless browser from a residential address would see more than this probe did, and some of the 403s are plausibly bot mitigation reacting to a datacenter IP, or an inherited default like the one described above, rather than a deliberate policy about agents. This probe measured none of those causes and the adverbs in earlier drafts were doing work the data did not support. The honest reading is that this measures how the web answers a cheap client, not how it answers every agent.

What would replace these figures: the same probe run monthly over a larger sample, from three IP classes, with and without a browser engine, and with enough repeats per cell to separate policy from noise. Until that exists, treat the numbers above as a first look and not as an index.

Reproduce it

The probe is a single Python file with no dependencies. Run it against your own sites. It writes a dated JSON file and prints the medians, the sites that flip between user agents, and a reminder that n is 1 per cell.

cd research && python3 probe.py

It writes probe-<today>.json into the directory you run it from and prints the medians, the sites that answer differently to each user agent, and a reminder that n is 1 per cell. Source: probe.py. The run behind this page: probe-2026-09-04.json. Both are CC0.

6Implement

32.0 KB as /implement.md, updated 2026-09-07, sha256 98f449e5f7a6

You need two things. A directory of markdown files, and something that decides which representation to return. Everything below is CC0.

Cloudflare Workers

This is the exact file txtfirst.com runs, injected into this page at build time from src/index.js. It cannot drift from what is deployed, because there is only one copy and this is it.

// txtfirst.com runs as a Cloudflare Worker with static assets.
// One HTML page for people at /. Everything else is for programs.
const UA = "TxtFirstProbe/0.1 (+https://txtfirst.com/probe; research on agent readability)";

export default {
  async fetch(request, env) {
    const res = await handle(request, env);
    record(env, request, res);
    return res;
  },
};

async function handle(request, env) {
    const url = new URL(request.url);

    // One canonical host. www redirects rather than serving a second copy.
    if (url.hostname === "www.txtfirst.com") {
      url.hostname = "txtfirst.com";
      return Response.redirect(url.toString(), 301);
    }

    if (url.pathname === "/api/check") return check(url, env, request);
    if (url.pathname === "/reads" || url.pathname === "/reads.md") return reads(request, env, url);

    const slug = url.pathname === "/" ? "/index" : url.pathname.replace(/\/$/, "");
    const isMdPath = url.pathname.endsWith(".md");
    const wantsMd = !isMdPath &&
      (request.headers.get("accept") || "").includes("text/markdown");

    // R2: the same URL, the source instead of the rendering.
    if (wantsMd) {
      const md = await env.ASSETS.fetch(new URL(slug + ".md", url.origin));
      if (md.ok) {
        return new Response(md.body, {
          headers: {
            "content-type": "text/markdown; charset=utf-8",
            "vary": "Accept",
            "cache-control": "public, max-age=300",
            "x-txtfirst": "0.1",
          },
        });
      }
    }

    // A section URL asked for by a browser goes to its anchor on the one page.
    // Only for slugs that exist as markdown, so an unknown path still 404s.
    if (!isMdPath && slug !== "/index" && !url.pathname.includes(".")) {
      const src = await env.ASSETS.fetch(new URL(slug + ".md", url.origin));
      if (src.ok) {
        return new Response(null, {
          status: 301,
          headers: {
            "location": "/#" + slug.slice(1),
            "link": `<${slug}.md>; rel="alternate"; type="text/markdown"`,
            "vary": "Accept",
            "x-txtfirst": "0.1",
          },
        });
      }
    }

    // ASSETS.fetch returns a Response. Re-wrap it with the response as init,
    // not as the body, or you serve the string "[object Response]".
    const upstream = await env.ASSETS.fetch(request);
    const res = new Response(upstream.body, upstream);
    if (isMdPath) {
      res.headers.set("content-type", "text/markdown; charset=utf-8");
    } else if (url.pathname.endsWith(".py") || url.pathname.startsWith("/cli/")) {
      // Source we publish as evidence should render, not download.
      res.headers.set("content-type", "text/plain; charset=utf-8");
    } else {
      res.headers.set("link", `<${slug}.md>; rel="alternate"; type="text/markdown"`);
    }
    res.headers.append("vary", "Accept");
    res.headers.set("x-txtfirst", "0.1");
    return res;
}

// ---------- who reads what ----------
// One row per response in Analytics Engine: what shape was served, to which kind of
// client. No IP, no cookie, no identifier. The question this site exists to answer is
// whether programs ask for the markdown, and this is the only place that can see it.
function family(ua) {
  const s = ua.toLowerCase();
  if (!s) return "none";
  if (s.includes("txtfirstprobe")) return "self";
  if (/gptbot|chatgpt|openai|oai-searchbot/.test(s)) return "openai";
  if (/claudebot|claude-user|claude-searchbot|anthropic/.test(s)) return "anthropic";
  if (/perplexity/.test(s)) return "perplexity";
  if (/googlebot|google-extended|gemini/.test(s)) return "google";
  if (/bingbot|copilot/.test(s)) return "microsoft";
  if (/bot|crawl|spider|fetch|scrap|preview|monitor/.test(s)) return "other-bot";
  if (/curl|wget|python|httpx|go-http|node|java|ruby|php|libwww/.test(s)) return "cli";
  if (/mozilla/.test(s)) return "browser";
  return "other";
}
function shape(res, path) {
  const ct = res.headers.get("content-type") || "";
  if (path === "/api/check") return "check";
  if (res.status >= 300 && res.status < 400) return "redirect";
  // Status before content-type. The 404 page is served as text/html, so reading the
  // content-type first counted every vulnerability scanner as a page view: 185 of the
  // first 291 "html" rows were 404s, mostly /wp-login.php and /.env. A miss is not a read.
  if (res.status >= 400) return "miss";
  if (ct.includes("text/markdown")) return "md";
  if (ct.includes("text/html")) return "html";
  if (ct.includes("json")) return "json";
  if (ct.includes("text/plain")) return "txt";
  return "other";
}
function record(env, request, res) {
  try {
    if (!env.READS) return;
    const url = new URL(request.url);
    const ua = request.headers.get("user-agent") || "";
    const kind = shape(res, url.pathname);
    env.READS.writeDataPoint({
      blobs: [kind, url.pathname.slice(0, 96), family(ua), ua.slice(0, 250),
              (request.headers.get("accept") || "").slice(0, 80)],
      doubles: [res.status],
      indexes: [kind],
    });
  } catch { /* measuring must never break serving */ }
}

// ---------- /api/check?host=<hostname> ----------
// The probe from /measure, as a service. Two fetches at most, as a plainly identified
// client, honouring robots.txt for TxtFirstProbe. Cached 24h per host so a hostname
// is asked at most once a day no matter how many callers ask.

const HOST_RE = /^(?=.{1,253}$)(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z]{2,63}$/;

function json(obj, status = 200, extra = {}) {
  return new Response(JSON.stringify(obj, null, 1), {
    status,
    headers: { "content-type": "application/json; charset=utf-8", "vary": "Accept",
               "cache-control": "public, max-age=3600", "x-txtfirst": "0.1", ...extra },
  });
}

async function check(url, env, request) {
  // A browser (the form on the one page) gets a rendering. A program gets the JSON.
  const accept = request.headers.get("accept") || "";
  const reply = accept.includes("text/html") && !accept.includes("application/json") ? page : json;
  const host = (url.searchParams.get("host") || "").trim().toLowerCase()
    .replace(/^https?:\/\//, "").replace(/\/.*$/, "");
  if (!HOST_RE.test(host) || /(^|\.)(localhost|local|internal|arpa|home|lan)$/.test(host)) {
    return reply({ error: "host must be a public hostname: no scheme, path, port or IP" }, 400);
  }
  const key = "check:" + host;
  const self = host === "txtfirst.com";
  const hit = env.CACHE && !self ? await env.CACHE.get(key) : null;
  if (hit) return reply(JSON.parse(hit), 200, { "x-cache": "hit" });

  const out = { host, checked: new Date().toISOString(), user_agent: UA, robots: null, home: null };

  // A Worker cannot reach its own zone through the public URL (Cloudflare answers 522),
  // so the self check reads the assets directly. Everything else goes out the front.
  const fetchFn = self ? (t) => env.ASSETS.fetch(new URL(new URL(t).pathname, url.origin)) : fetch;

  const r = await probe("https://" + host + "/robots.txt", fetchFn);
  out.robots = { status: r.status, bytes: r.bytes };
  // Same reading as Python's robotparser, which the published probe uses: 401/403
  // means disallow everything, another 4xx means allow, unreachable means allow.
  let allowed = true;
  if (r.status === 401 || r.status === 403) allowed = false;
  else if (r.status === 200) allowed = robotsAllow(r.text, "txtfirstprobe", "/");
  out.robots.allows_probe = allowed;

  if (allowed) {
    const h = await probe("https://" + host + "/", fetchFn);
    const text = stripped(h.text);
    out.home = {
      status: h.status, html_bytes: h.bytes, text_bytes: text.length,
      ratio: text.length ? +(h.bytes / text.length).toFixed(1) : null,
      readable: h.status === 200 && text.length >= 1500,
    };
    out.verdict = out.home.readable ? "readable"
      : (h.status === 200 || h.status === 202) ? "no content" : "blocked";
  } else {
    out.verdict = "robots-disallow";
  }
  out.threshold = "readable means status 200 and at least 1500 bytes of text after stripping markup";

  // Cache only a real answer from the host. A timeout or a 5xx describes this minute,
  // not the site, and must not be remembered for a day.
  const solid = (s) => typeof s === "number" && s < 500;
  const cacheable = !self && solid(r.status) && (!out.home || solid(out.home.status));
  if (env.CACHE && cacheable) await env.CACHE.put(key, JSON.stringify(out), { expirationTtl: 86400 });
  return reply(out, 200, { "x-cache": cacheable ? "miss" : "uncached" });
}

// The same result as HTML, for a person who typed a hostname into the form.
// Same numbers, same JSON underneath, no script.
function esc(s) { return String(s).replace(/[&<>"]/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;" }[c])); }
function page(obj, status = 200, extra = {}) {
  const v = obj.verdict, h = obj.home;
  const line = obj.error ? esc(obj.error)
    : v === "readable" ? `<b>${esc(obj.host)}</b> answers a plainly identified program: ${h.text_bytes} bytes of text inside ${h.html_bytes} bytes of HTML, ${h.ratio} to 1.`
    : v === "no content" ? `<b>${esc(obj.host)}</b> answered ${h.status} but only ${h.text_bytes} bytes of text survived stripping the markup. The page is there, the content is not.`
    : v === "blocked" ? `<b>${esc(obj.host)}</b> refused the program with ${esc(h.status)}. A browser would probably get the page.`
    : `<b>${esc(obj.host)}</b> disallows this probe in robots.txt, so nothing was measured.`;
  const body = `<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>check ${esc(obj.host || "")}</title>
<style>body{max-width:74ch;margin:2.2rem auto;padding:0 1.4rem;font:15px/1.72 ui-monospace,Menlo,Consolas,monospace;background:#fbfbf8;color:#16150f}
a{color:#8a3a12}pre{background:#f2f0e6;padding:1rem;overflow-x:auto;font-size:.88em}.dim{color:#6d6a5c}
@media(prefers-color-scheme:dark){body{background:#12120f;color:#deddd2}pre{background:#1c1c17}a{color:#e0854a}.dim{color:#8b8878}}</style></head>
<body><p><a href="/">txtfirst</a> <span class="dim">/ check</span></p>
<h1>${obj.error ? "not checked" : esc(v)}</h1>
<p>${line}</p>
<p class="dim">${obj.error ? "" : "readable means status 200 and at least 1500 bytes of text after stripping markup. The probe sends two GETs at most, identifies itself, honours robots.txt, and is answered from cache for a day. What it sends and how to block it: <a href=\"/#probe\">probe</a>."}</p>
<p>The same answer as a program sees it, at <a href="/api/check?host=${esc(obj.host || "")}">this URL</a> with <code>Accept: application/json</code>:</p>
<pre>${esc(JSON.stringify(obj, null, 1))}</pre>
<p><a href="/">Check another</a></p></body></html>`;
  return new Response(body, { status, headers: { "content-type": "text/html; charset=utf-8",
    "cache-control": "public, max-age=3600", "vary": "Accept", "x-txtfirst": "0.1", ...extra } });
}

async function probe(target, fetchFn = fetch) {
  const ctl = new AbortController();
  const timer = setTimeout(() => ctl.abort(), 12000);
  try {
    const res = await fetchFn(target, {
      headers: { "user-agent": UA, "accept": "text/html,*/*" },
      redirect: "follow", signal: ctl.signal,
    });
    const text = await res.text();
    return { status: res.status, bytes: new TextEncoder().encode(text).length, text };
  } catch (e) {
    return { status: e.name === "AbortError" ? "timeout" : "error", bytes: 0, text: "" };
  } finally {
    clearTimeout(timer);
  }
}

function stripped(html) {
  return html
    .replace(/<(script|style|noscript|svg|template|iframe)[^>]*>[\s\S]*?<\/\1>/gi, " ")
    .replace(/<!--[\s\S]*?-->/g, " ")
    .replace(/<[^>]+>/g, " ")
    .replace(/&nbsp;/g, " ")
    .replace(/\s+/g, " ")
    .trim();
}

// Minimal robots.txt reader: does the most specific matching group allow `path`?
// Only what is needed to honour a Disallow aimed at this probe. Not a full RFC 9309.
function robotsAllow(txt, agent, path) {
  let groups = [], cur = null;
  for (let line of txt.split(/\r?\n/)) {
    line = line.replace(/#.*/, "").trim();
    const m = line.match(/^([a-z-]+)\s*:\s*(.*)$/i);
    if (!m) continue;
    const k = m[1].toLowerCase(), v = m[2].trim();
    if (k === "user-agent") {
      if (!cur || cur.rules.length) { cur = { agents: [], rules: [] }; groups.push(cur); }
      cur.agents.push(v.toLowerCase());
    } else if (cur && (k === "allow" || k === "disallow")) {
      cur.rules.push([k, v]);
    }
  }
  const pick = groups.find(g => g.agents.some(a => a !== "*" && agent.includes(a)))
            || groups.find(g => g.agents.includes("*"));
  if (!pick) return true;
  let best = null;
  for (const [k, v] of pick.rules) {
    if (v === "" ) { if (k === "disallow" && best === null) best = ["allow", ""]; continue; }
    if (path.startsWith(v) && (best === null || v.length > best[1].length)) best = [k, v];
  }
  return best === null || best[0] === "allow";
}

// ---------- /reads ----------
// What this site is asked for, and by what. Generated from the same dataset the Worker
// writes to, cached an hour, published because a site making an argument about
// measurement should show its own numbers rather than describe them.
const ACCOUNT = "38b9d6052960175311a8a3ecb2012e29";
// Counting starts here, not at the first row ever written. Before this moment a 404
// served as text/html was recorded as a page view, so the earlier rows cannot be
// compared with the later ones and are not shown. Moving this line back would make the
// totals larger and wrong.
// 06:29:46 UTC is the first row the corrected Worker wrote, found by asking the dataset
// for the earliest row carrying a label the old code could not produce, rather than by
// converting the deploy time in my head, which put the floor in the future and returned
// a confident zero.
const SINCE = "2026-09-07 06:29:46";

async function ae(env, sql) {
  const res = await fetch(`https://api.cloudflare.com/client/v4/accounts/${ACCOUNT}/analytics_engine/sql`, {
    method: "POST",
    headers: { authorization: `Bearer ${env.AE_READ_TOKEN}`, "content-type": "text/plain" },
    body: sql + " FORMAT JSON",
  });
  const body = await res.json();
  if (!body.data) throw new Error("analytics query failed");
  return body.data;
}

function md_table(rows, cols) {
  if (!rows.length) return "(nothing recorded yet)\n";
  const w = cols.map((c) => Math.max(c.length, ...rows.map((r) => String(r[c] ?? "").length)));
  const pad = (v, i) => String(v ?? "").padEnd(w[i]);
  return "| " + cols.map(pad).join(" | ") + " |\n"
       + "|" + w.map((n) => "-".repeat(n + 2)).join("|") + "|\n"
       + rows.map((r) => "| " + cols.map((c, i) => pad(r[c], i)).join(" | ") + " |").join("\n") + "\n";
}

async function readsMarkdown(env) {
  const base = `FROM txtfirst_reads WHERE timestamp > toDateTime('${SINCE}')`;
  const n = "SUM(_sample_interval) AS n";
  const [shapes, agents, paths] = await Promise.all([
    ae(env, `SELECT blob1 AS shape, blob3 AS client, ${n} ${base} AND blob3 != 'self' GROUP BY shape, client ORDER BY n DESC LIMIT 30`),
    ae(env, `SELECT blob3 AS client, blob4 AS agent, blob1 AS shape, ${n} ${base} AND blob3 IN ('openai','anthropic','perplexity','google','microsoft','other-bot') GROUP BY client, agent, shape ORDER BY n DESC LIMIT 20`),
    ae(env, `SELECT blob2 AS path, blob1 AS shape, ${n} ${base} AND blob1 IN ('md','html','txt','json') GROUP BY path, shape ORDER BY n DESC LIMIT 15`),
  ]);
  const num = (r) => ({ ...r, n: Number(r.n) });
  const S = shapes.map(num), A = agents.map(num), P = paths.map(num);
  const sum = (rows, f = () => true) => rows.filter(f).reduce((a, r) => a + r.n, 0);
  const md = sum(S, (r) => r.shape === "md");
  const html = sum(S, (r) => r.shape === "html");
  const miss = sum(S, (r) => r.shape === "miss");
  const total = sum(S);
  const today = new Date().toISOString().slice(0, 10);
  const was = (k) => k === 1 ? "was a miss" : "were misses";
  return `---
title: What this site is asked for
desc: Every request txtfirst.com has answered since its own measurement was corrected, by shape and by client. Generated from live data, not written by hand.
---

# What this site is asked for

Generated ${today}, counting every response since ${SINCE} UTC. This page is not written by anyone. The
Worker records one row per response, with no IP address, no cookie and no identifier:
the shape it served, the path, the client family, the user agent and the Accept header.
That is the whole dataset.

Measurement began on 6 September 2026, two days after the domain went live, and the
count here starts a day later than that. The first day of rows classified a 404 served
as an HTML page as though it were a page view, which turned 185 vulnerability scanners
into readers. To be exact about that number: 185 requests from 22 different clients,
seven of which were for a favicon, which is not a scanner but is not a reader either.
That is fixed, and the mislabelled rows are excluded rather than quietly included, so
these numbers are small. They are published anyway, because a site arguing
that the web should be measurable should be measurable itself, including when its own
first measurement was wrong.

## The one number

Of **${total} requests** that were not this site's own probe, **${md} asked for markdown**
and **${html} were answered with the HTML page**. **${miss} ${was(miss)}**. A miss is a
request for something that is not here, and most of them are vulnerability scanners
looking for \`/wp-login.php\` and \`/.env\`, which arrive at any domain within hours of it
existing and are not readers. Misses are counted apart rather than as traffic.

## Shape by client

\`browser\` is a client sending a browser user agent, which includes any program
pretending to be one. \`cli\` is curl, wget or a plain HTTP library. \`other-bot\` is a
declared crawler that is not one of the named model builders. A shape of \`miss\` is a
404 or a 405.

${md_table(S, ["shape", "client", "n"])}
## Declared crawlers

Every request from a client that names itself as a crawler, with the shape it received.
If this table is empty, or holds only crawlers that are not model builders, that is
itself the finding: publishing in a machine readable format does not summon anyone.

${md_table(A, ["client", "agent", "shape", "n"])}
## Paths people and programs actually reach

Misses excluded, so this is what was found rather than what was guessed at.

${md_table(P, ["path", "shape", "n"])}
## What this cannot tell you

A user agent is a claim, not an identity, and anything can send any string. This page
cannot separate a person clicking a \`.md\` link from a scraper wearing a Chrome string.
Analytics Engine samples under load, so counts are estimates. Nothing
here is a visit count and none of it identifies anyone.

Check any site yourself with the checker this site publishes, which needs nothing
installed beyond Node:

\`\`\`
curl -s https://txtfirst.com/cli/index.js > txtfirst.js
node txtfirst.js check yourdomain.com
\`\`\`
`;
}

async function reads(request, env, url) {
  if (!env.AE_READ_TOKEN) return new Response("not configured", { status: 503 });
  // The key carries a version so that changing the prose below invalidates the cache.
  // Without it a corrected sentence stays unpublished for an hour and reads like a
  // deploy that did not land. Bump it whenever readsMarkdown changes.
  const key = "reads:md:v3";
  let body = env.CACHE ? await env.CACHE.get(key) : null;
  if (!body) {
    try { body = await readsMarkdown(env); } catch (e) {
      return new Response("The measurement is unavailable right now, which is a failure of this page and not a report that nothing was read.\n",
        { status: 503, headers: { "content-type": "text/plain; charset=utf-8" } });
    }
    if (env.CACHE) await env.CACHE.put(key, body, { expirationTtl: 3600 });
  }
  const accept = request.headers.get("accept") || "";
  const wantsHtml = url.pathname === "/reads" && accept.includes("text/html") && !accept.includes("text/markdown");
  const h = { "cache-control": "public, max-age=600", "vary": "Accept", "x-txtfirst": "0.1",
              "link": '</reads.md>; rel="alternate"; type="text/markdown"' };
  if (!wantsHtml) return new Response(body, { headers: { ...h, "content-type": "text/markdown; charset=utf-8" } });
  return new Response(mdPage(body), { headers: { ...h, "content-type": "text/html; charset=utf-8" } });
}

// The smallest markdown rendering that is honest: enough for headings, tables, code
// and bold. The page is data, and the source is one header away at /reads.md.
function mdPage(src) {
  const esc = (t) => t.replace(/[&<>]/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;" }[c]));
  const inline = (t) => esc(t)
    .replace(/`([^`]+)`/g, "<code>$1</code>")
    .replace(/\*\*([^*]+)\*\*/g, "<strong>$1</strong>");
  const body = src.replace(/^---[\s\S]*?---\n/, "");
  const out = [];
  let rows = [];
  const flush = () => {
    if (!rows.length) return;
    const cells = rows.filter((r) => !/^\|[-|\s]+\|$/.test(r))
      .map((r) => r.slice(1, -1).split("|").map((c) => c.trim()));
    const head = cells.shift() || [];
    out.push("<table><thead><tr>" + head.map((c) => `<th>${inline(c)}</th>`).join("") + "</tr></thead><tbody>"
      + cells.map((r) => "<tr>" + r.map((c) => `<td>${inline(c)}</td>`).join("") + "</tr>").join("") + "</tbody></table>");
    rows = [];
  };
  for (const line of body.split("\n")) {
    if (/^\s*\|.*\|\s*$/.test(line)) { rows.push(line.trim()); continue; }
    flush();
    const h = line.match(/^(#{1,3})\s+(.*)$/);
    if (h) out.push(`<h${h[1].length}>${inline(h[2])}</h${h[1].length}>`);
    else if (line.trim()) out.push(`<p>${inline(line)}</p>`);
  }
  flush();
  return `<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>What txtfirst.com is asked for</title>
<meta name="description" content="The requests txtfirst.com answers, by shape and by client, generated from live data.">
<link rel="alternate" type="text/markdown" href="https://txtfirst.com/reads.md">
<style>body{max-width:74ch;margin:2.2rem auto;padding:0 1.4rem 5rem;font:15px/1.72 ui-monospace,Menlo,Consolas,monospace;background:#fbfbf8;color:#16150f}
a{color:#8a3a12}h1{font-size:1.5rem;letter-spacing:-.03em;border-bottom:1px solid #d9d5c4;padding-bottom:.9rem}
h2{font-size:1.05rem;margin:2.4rem 0 .6rem}h2::before{content:"## ";color:#6d6a5c;font-weight:400}
code{background:#f2f0e6;padding:.1em .35em}table{border-collapse:collapse;width:100%;font-size:.88em;display:block;overflow-x:auto;margin:0 0 1.2rem}
th,td{text-align:left;padding:.3rem 1rem .3rem 0;border-bottom:1px solid #d9d5c4;white-space:nowrap}th{color:#6d6a5c;font-weight:400}
@media(prefers-color-scheme:dark){body{background:#12120f;color:#deddd2}a{color:#e0854a}code,th,td{border-color:#33322a}code{background:#1c1c17}h1{border-color:#33322a}th{color:#8b8878}}</style>
</head><body><p><a href="/">txtfirst</a> <span style="color:#6d6a5c">/ reads</span></p>
${out.join("\n")}
<p style="color:#6d6a5c">This page as its source: <a href="/reads.md">/reads.md</a></p>
</body></html>`;
}

With wrangler.toml:

name = "txtfirst"
main = "src/index.js"
compatibility_date = "2026-04-01"

[assets]
directory = "./dist"
binding = "ASSETS"
run_worker_first = true
html_handling = "auto-trailing-slash"
not_found_handling = "404-page"

run_worker_first matters. Without it the static asset is served before your code runs and the negotiation never happens.

Cloudflare Pages

Same logic as a Pages Function. Note the isMdPath branch: without it, /path.md is served with whatever content type Pages guesses, and R2 says it MUST be text/markdown.

// functions/_middleware.js
export async function onRequest({ request, next }) {
  const url = new URL(request.url);
  const slug = url.pathname === "/" ? "/index" : url.pathname.replace(/\/$/, "");
  const isMdPath = url.pathname.endsWith(".md");
  const wantsMd = !isMdPath &&
    (request.headers.get("accept") || "").includes("text/markdown");

  if (wantsMd) {
    const md = await fetch(new URL(slug + ".md", url.origin));
    if (md.ok) {
      return new Response(md.body, {
        headers: {
          "content-type": "text/markdown; charset=utf-8",
          "vary": "Accept",
          "cache-control": "public, max-age=300",
        },
      });
    }
  }

  // next() already returns a Response. Re-wrap it with the response as init,
  // not as the body, or you will serve the string "[object Response]".
  const upstream = await next();
  const res = new Response(upstream.body, upstream);
  if (isMdPath) {
    res.headers.set("content-type", "text/markdown; charset=utf-8");
  } else {
    res.headers.set("link", `<${slug}.md>; rel="alternate"; type="text/markdown"`);
  }
  res.headers.append("vary", "Accept");
  return res;
}

nginx

If you already build both files, negotiation is a map and a try_files.

map $http_accept $md_pref { default 0; "~*text/markdown" 1; }

location / {
  if ($md_pref) { rewrite ^/(.*)$ /$1.md last; }
  try_files $uri $uri.html $uri/index.html =404;
  add_header Link "<$uri.md>; rel=\"alternate\"; type=\"text/markdown\"" always;
  add_header Vary "Accept" always;
}

location ~ \.md$ {
  default_type "text/markdown; charset=utf-8";
}

Apache

AddType text/markdown;charset=utf-8 .md
RewriteEngine On
RewriteCond %{HTTP:Accept} text/markdown
RewriteCond %{REQUEST_FILENAME}.md -f
RewriteRule ^(.*)$ $1.md [L]
Header always set Vary "Accept"

Vercel

Same idea in middleware.ts. Rewrite to ${pathname}.md when the Accept header asks for markdown, and set the Link header on everything else.

Astro

Astro already has a middleware hook and a content collection whose entries are markdown files, which makes it the most natural fit of any framework here. Untested by us: read it as a sketch, not a recipe.

// src/middleware.ts
import { defineMiddleware } from "astro:middleware";
export const onRequest = defineMiddleware(async ({ request, url }, next) => {
  const accept = request.headers.get("accept") || "";
  const wantsMd = url.pathname.endsWith(".md") ||
    (accept.includes("text/markdown") && !url.pathname.includes("."));
  if (wantsMd) {
    const slug = url.pathname.replace(/\.md$/, "").replace(/\/$/, "") || "/index";
    // On Cloudflare do not fetch your own origin (522, see the Workers file above):
    // use locals.runtime.env.ASSETS.fetch, or read getEntry(...).body from astro:content.
    const md = await fetch(new URL(`/_md${slug}.md`, url.origin)); // emitted at build
    if (md.ok) return new Response(md.body, { headers: {
      "content-type": "text/markdown; charset=utf-8", "vary": "Accept" } });
  }
  const res = await next();
  res.headers.set("link", `<${url.pathname.replace(/\/$/, "")}.md>; rel="alternate"; type="text/markdown"`);
  res.headers.append("vary", "Accept");
  return res;
});

At build time, copy every content collection entry's raw markdown to public/_md/<slug>.md unchanged. The collection file is the source. The rendered page is derived from it. That satisfies R1 without any second copy to maintain: the _md tree is regenerated on every build and never edited.

WordPress

WordPress stores HTML, not markdown, so a plugin can only serve a derived markdown, and the honest name for that is a compatibility layer. The sketch covers the negotiation half of R2. The Link header on the HTML response and the R3 manifest are yours to add. It does not meet R1, because the post editor is the source and the markdown is an export of it. Say so on the site rather than claim compliance. Untested sketch:

// txtfirst.php, drop into wp-content/mu-plugins/
add_action('template_redirect', function () {
  $accept = $_SERVER['HTTP_ACCEPT'] ?? '';
  $wants  = str_contains($accept, 'text/markdown') || str_ends_with($_SERVER['REQUEST_URI'], '.md');
  if (!$wants || !is_singular()) return;
  $post = get_queried_object();
  header('Content-Type: text/markdown; charset=utf-8');
  header('Vary: Accept');
  echo "---\ntitle: " . $post->post_title . "\nupdated: " . $post->post_modified_gmt . "\n---\n\n";
  echo "# " . $post->post_title . "\n\n" . html_to_markdown(apply_filters('the_content', $post->post_content));
  exit;
});
add_action('wp_head', function () {
  if (is_singular()) echo '<link rel="alternate" type="text/markdown" href="' . rtrim(get_permalink(), '/') . '.md">';
});

html_to_markdown is any HTML to Markdown converter. The .md suffix needs a rewrite rule so WordPress does not 404 it before the hook runs.

Plain HTML, no framework

If your site is a folder of HTML files, the smallest compliant version is the one this site ran on its first day: keep the markdown files next to the HTML, generate the HTML from them with a script, and put the negotiation in whatever serves the folder (the nginx and Apache blocks above). build.py on this site is a working example with no dependencies.

Check it, from anywhere

The checker is a single file with no dependencies. It makes at most seven GET requests to one origin, identifies itself as txtfirst-cli, asks robots.txt first, writes nothing and sends nothing anywhere.

curl -s https://txtfirst.com/cli/index.js > txtfirst.js

node txtfirst.js check yourdomain.com
node txtfirst.js check yourdomain.com --path /blog/a-post
node txtfirst.js check yourdomain.com --json      # for CI

It answers two questions in order. First, whether the origin answers a client that is not pretending to be a browser, which most do not and which nothing else can fix. Second, whether it meets the four requirements above. It exits 0 when a site is conformant, 1 when it is not, 2 on bad input and 3 when it could not measure because robots.txt disallows it, so it can gate a deploy without a site that refuses the checker being read as a site that passed.

The source is at /cli/index.js, which is the same file the command downloads. Running it against txtfirst.com should return eight of eight requirement checks and two of two access checks, and if it ever does not, this page is wrong rather than your reading of it.

A running system, not a sketch

Three news sites sharing one chassis (Cloudflare Pages Functions, KV, TypeScript) adopted the layer on 6 September 2026. Each article body was already one markdown document in KV, which is why the patch is small. The patch added the .md suffix, Accept negotiation, Link and <link> advertising, a /changes.json feed and the manifest, and moved a glossary tooltip's definition out of the article text into an attribute so a client that strips markup no longer reads it mid sentence. Applied by a script with an assert on every load-bearing edit, so a drifted copy fails loudly instead of being half patched: chassis_kv.py. Verified live by check_chasis.sh, fourteen checks per site.

The build step

Any static site generator already does this, you are just changing which output you consider canonical. If you want the smallest possible version, this site's build is a single Python file with no dependencies that walks content/, writes slug.md byte identical to the source and slug.html rendered from it, then emits robots.txt, llms.txt, sitemap.xml and /.well-known/txtfirst.json.

python3 build.py          # build
python3 build.py --check  # self test the renderer

Verify yourself

curl -sI https://example.com/page | grep -i '^link:'
curl -s -H "Accept: text/markdown" https://example.com/page | head -3
curl -s https://example.com/page.md | head -3
diff <(curl -s -H "Accept: text/markdown" https://example.com/page) \
     <(curl -s https://example.com/page.md) && echo "R2 ok"
curl -s https://example.com/.well-known/txtfirst.json | head -5

If the diff is empty and the JSON parses, you meet R2 and R3. R1 is a discipline, not a header, and nobody can check it for you. R4 needs a human to look at your images once.

7TxtFirstProbe

2.4 KB as /probe.md, updated 2026-09-04, sha256 6b1837492cd9

If you found this page in your logs, something here requested your homepage. This page exists so you can see exactly what, without asking.

It sends two requests, and one of them is disguised

The whole point of the measurement is to compare what a site returns to a browser against what it returns to an identified program. That means the probe requests your homepage twice, once as itself and once wearing a current Chrome string:

TxtFirstProbe/0.1 (+https://txtfirst.com/probe; research on agent readability)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36

The second string is not honest, and a site running an argument about honest identification should say so on the record rather than let you find it in the source. It is there because the comparison is the finding. Without it there is nothing to compare against and no way to tell a site that blocks agents from a site that blocks everything.

What it does with them

Two GET requests to your homepage, run once. No JavaScript execution, no cookies, no retries, no crawling of any other path, 25 second timeout. It records the HTTP status, the byte size of the response, and the byte size of the text left after stripping markup. It does not store your content, only those three numbers per request. The results are published at measure and the raw output is public.

How to block it

robots.txt is checked before either request and both are skipped if you disallow them. This is done by urllib.robotparser in the probe source, not by a promise on a page.

User-agent: TxtFirstProbe
Disallow: /

A disallowed site is recorded as robots-disallow and carries no measurement. Nothing is inferred from a block and no site is named for blocking.

Why it exists

To answer one question with a number instead of an opinion: when a program that is not a browser asks a website for its front page, what happens. The answer so far is that most of the time nothing useful happens, which is the argument this whole site is making. The sample is small and the limitations say so.

A human

Write to hello at this domain. Removal requests are honoured without discussion and do not require a reason.

8If you are an agent

2.4 KB as /agent.md, updated 2026-09-06, sha256 0aa6d675ba47

You are reading a site built to be read by you. This section tells you how, in the order you will need it.

Read

Every section of this site is one markdown file. Fetch it directly and skip the HTML:

GET https://txtfirst.com/spec.md
GET https://txtfirst.com/spec            Accept: text/markdown
GET https://txtfirst.com/api/spec.json   the same markdown inside JSON, with sha256

The list of everything, with sizes and hashes, is at /api/pages.json. The whole site as one file is /llms-full.txt. Its size is the sum of bytes in that list, so this page does not quote a number that rots.

Do not re-read

Fetch /changes.json and compare sha256 per slug against what you last saw. A section's updated date moves only when its bytes move. If nothing moved, you are done and you have spent one request.

Check another site

To learn whether a hostname answers a plainly identified program, ask:

GET https://txtfirst.com/api/check?host=example.com

You get back the status and byte counts for /robots.txt and /, whether their robots allows this probe, and a verdict: readable, no content, blocked or robots-disallow. A host that answered is not asked again for a day. A timeout or a server error is not cached, so a host that was down is asked again next time. The probe fetches at most two URLs, identifies itself, and honours robots.txt. Pass a public hostname only: no scheme, path, port or IP address. The same URL answers with HTML when the client accepts text/html and not application/json, which is how the form on the one page works. Send Accept: application/json to be sure of JSON.

Quote

When you cite a number from measure, cite the page and the date. The figures come from one run on one day and the page says so. Do not present them as current.

Do not

• Do not fetch the HTML at / to extract text. The markdown is the text.
• Do not use /api/check to test a host repeatedly. A host that answered is served from cache for a day. It has already been asked.
• Do not invent fields for a Fact Block. The eight fields are listed on dialect, in order, and a block with a ninth is not one.

Contact

The mailbox is hello at this domain. It reads plain text.

9About

1.5 KB as /about.md, updated 2026-09-06, sha256 fff15622f638

txtfirst is an open proposal for how content sites should be built when a large share of their readers are programs. It is maintained by an independent developer working on travel, hospitality and search, and it started because a measurement of twenty five hotel websites came back worse than expected.

What this is not

It is not a product, a company, a SaaS, or a standards body. Nothing here is for sale and there is nothing to sign up for. There is no certification and there will not be one, because a compliance badge nobody audits is worth less than a curl command anyone can run.

Funding and conflicts

Unfunded. No sponsors, no affiliate links on this domain, no advertising. The author builds and operates content sites that would benefit if txtfirst were adopted widely, which is a conflict of interest and is disclosed here rather than buried.

Licence

Everything on this domain, including the spec, the reference implementation and the research data, is released under CC0 1.0. Use it commercially, fork it, rename it, ship it without credit. If the ideas survive under someone else's name, the proposal worked.

Contributing

The spec is version 0.1 and the open questions are listed at the bottom of spec. Disagreement is more useful than agreement right now, particularly on R2 and on whether R4 is too strict to be adopted.

Reach the author through contact.

10Contact

1.4 KB as /contact.md, updated 2026-09-06, sha256 0c9f7799b63d

Write to hello at this domain.

What gets a reply

• You implemented txtfirst on a real site and something in the spec did not survive contact with reality. This is the most useful message you can send.
• You disagree with a requirement and can say which one and why.
• You ran the probe from measure on a different sample and got a different answer. Send the JSON.
• You are building a client that would consume this and want something changed before it is too late to change it.

What does not

Sales, guest posts, link exchanges, and offers to write about this site. There is no advertising here and there will not be.

Reporting a problem with this site

If a page renders wrong, the markdown is the source of truth. Send the URL and its .md twin and say which one is correct.

Machine readable

{
  "contact": "https://txtfirst.com/contact",
  "mailbox": "hello, at this domain",
  "prefers": ["text/markdown", "text/plain"],
  "index": "https://txtfirst.com/.well-known/txtfirst.json"
}

An agent reading this will notice the mailbox is not written as an address. That is a deliberate exception, and it is the one place on this site where being fully machine-readable loses to not feeding an address harvester. If you are a person, the first line of this page tells you what to do.