Does llms.txt help SEO? What it does, and when to add one

No, not for search rankings or AI citations. llms.txt is a proposed convention, not a standard any search engine has adopted. Google says Google Search ignores llms.txt files, and no major AI search product has said it uses them to pick sources. Where llms.txt does help is documentation: coding agents and IDE tools can read it to find the right docs page fast. So add one if you ship developer docs, skip it (or spend five minutes on it) if you run a marketing site, and put your real effort into crawler access, server-rendered HTML and indexing.

If your coding agent told you to add llms.txt "for SEO", this page explains what the file actually does, when it is worth having, and what to fix instead.

#What llms.txt is

llms.txt is a Markdown file at the root of a site (/llms.txt) that gives a language model a short summary of the site and a curated list of links. Jeremy Howard proposed it on September 3, 2024, at llmstxt.org. The idea: a model working with a small context window shouldn't have to crawl and clean up HTML pages full of navigation and scripts. It reads one short file and knows where to go.

The proposal defines a simple structure, in this order:

• An H1 with the project or site name. This is the only required part.
• An optional blockquote with a one-paragraph summary.
• Optional paragraphs or lists with more detail (no headings).
• Zero or more H2 sections, each a list of links in the form [name](url): notes. An H2 named Optional holds links an agent can skip when it needs a shorter context.

The proposal also suggests offering a clean Markdown copy of each page at the same URL plus .md. Many docs sites go one step further and publish /llms-full.txt, the full text of every docs page in one file. That second file isn't part of the llmstxt.org spec, but it is common: Next.js and Mintlify-hosted docs both serve one.

#Does llms.txt help SEO?

Look at what the companies that run search and answer engines have actually said, and what the traffic data shows.

Google: ignored by Search

Google's guide to optimizing for its generative AI features says you don't need machine-readable files, AI text files, markup or Markdown to appear in Google Search, "as Google Search itself doesn't use them." A note added in June 2026 says it is fine to maintain llms.txt for other services, and that doing so "won't harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them." The AI features page says the same for AI Overviews and AI Mode: no new AI text files or markup are needed, and there are no additional technical requirements. (More in our AI Overviews guide.)

Earlier, in April 2025, Google's John Mueller wrote on Reddit that, as far as he knew, none of the AI services had said they were using llms.txt, and compared it to the keywords meta tag: a file where a site owner claims what the site is about. Google stopped using the keywords meta tag for ranking long ago.

Other AI search products: no confirmed use

OpenAI, Anthropic and Perplexity document their search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and how robots.txt controls them. None of them says it uses llms.txt to decide which pages to show or cite in answers. Several of them publish an llms.txt for their own developer docs, which tells you who the file is really for.

The traffic data: almost nobody fetches it

Ahrefs looked at 137,210 domains in its web analytics product in May 2026. About 38,000 of them had a valid llms.txt. 97% of those files got zero requests that month, and the roughly 1,100 files that were fetched received about 22,000 requests in total, 96% of them from bots. Ahrefs notes its sample leans toward technical, SEO-aware sites.

#Where llms.txt does help

The audience that reads llms.txt today is coding agents and developer tools, not search engines. A developer asks Claude Code, Codex or Cursor to integrate your API. The agent (or the developer) fetches your /llms.txt, sees a clean list of guides and reference pages, and opens the right one instead of guessing from a search result. With /llms-full.txt, a tool can load all your docs in one request.

That is why the sites with the most useful llms.txt files are documentation sites. Next.js, Stripe, Vercel, OpenAI and Anthropic all serve one for their docs. Mintlify generates both llms.txt and llms-full.txt automatically for docs it hosts.

Two smaller reasons to add one anyway. It is cheap: a static text file, no dependencies. And Lighthouse's experimental "Agentic browsing" category (separate from its SEO score) has an llms.txt audit. That audit fails only when fetching the file causes a server error; a 404 is marked not applicable. So a missing file costs you nothing there, but a broken one does.

#Add it or skip it, by site type

If you already have one and it is stale, either update it from code (see below) or delete it. An llms.txt that lists dead URLs or wrong prices is worse than none for the agents that do read it.

#A minimal llms.txt

Keep it short and honest: what the product is, the few pages that answer most questions, and the facts an agent tends to get wrong (prices, limits, install command). Absolute URLs are safest.

# Acme

> Acme is a project planning API for small teams. REST and TypeScript SDK, free tier up to 3 projects.

- Install: `npm install @acme/sdk` (Node 20+)
- Auth: API keys from the dashboard, sent as a Bearer token.

## Docs

- [Quickstart](https://acme.com/docs/quickstart): create a key and make a first request
- [API reference](https://acme.com/docs/api): every endpoint, with request and response examples
- [Webhooks](https://acme.com/docs/webhooks): events, retries and signature checks

## Product

- [Pricing](https://acme.com/pricing): plans, limits and what counts as a project

## Optional

- [Changelog](https://acme.com/changelog): what changed, newest first

Leave out marketing copy, keyword lists and every blog post you ever wrote. The file isn't read for keywords, and a long list defeats its purpose.

#How to serve llms.txt in Next.js

Option 1: a static file in public/

Put the file at public/llms.txt and Next.js serves it at /llms.txt. This is the right choice for a small, hand-written file. The downside: it drifts when pages change, so add it to the list of things you update when you rename or remove a page.

Option 2: a route handler generated from your content

If your docs or pages live in code or a CMS, generate the file so it can't go stale. Create a folder named llms.txt with a route.ts inside it:

import { getDocs } from "@/lib/docs"; // your content source

// Prerender at build time (without Cache Components, GET handlers run per request).
export const dynamic = "force-static";

export async function GET() {
  const docs = await getDocs();
  const body = [
    "# Acme",
    "",
    "> Acme is a project planning API for small teams.",
    "",
    "## Docs",
    "",
    ...docs.map((d) => `- [${d.title}](https://acme.com/docs/${d.slug}): ${d.summary}`),
    "",
  ].join("\n");
  return new Response(body, {
    headers: { "content-type": "text/plain; charset=utf-8" },
  });
}

Don't keep both: a file in public/ and a route at the same path conflict. Since Next.js 15, GET route handlers aren't cached by default, so dynamic = "force-static" builds the file once. If you have Cache Components on, drop that line: a handler that reads no request data is prerendered anyway, and data fetching can go in a helper with "use cache".

The same pattern works for app/llms-full.txt/route.ts: concatenate the Markdown source of each docs page under its own heading.

#What actually gets you into AI answers

If the goal is to show up in ChatGPT search, Claude, Perplexity, AI Overviews or AI Mode, these matter far more than any text file:

1. Let the search crawlers in

   Allow Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot in robots.txt and at your CDN. OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended) is a separate choice and doesn't remove you from search. Check your rules with the free robots.txt tester, which covers the AI crawlers.

2. Put the content in the server HTML

   Many AI crawlers fetch raw HTML and don't run JavaScript. If your page is a client-rendered shell, they see nothing. Confirm with curl that your H1 and main answer are in the response.

3. Get indexed in Google and Bing

   AI answers lean on search indexes. Submit a sitemap in Search Console and Bing Webmaster Tools, and fix pages stuck in "Discovered, currently not indexed".

4. Answer the question clearly

   Put the direct answer near the top of the page, in plain text under a heading that matches the question. Add the specifics people need: numbers, steps, limits, prices. That is what gets quoted. Our ChatGPT search guide covers the OpenAI side.

#Check your setup

Four commands cover both the file and the things that matter more:

# llms.txt: 200 (or a clean 404), plain text, never a 500
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://your-site.com/llms.txt

# Every link in it still resolves
curl -s https://your-site.com/llms.txt | grep -oE 'https?://[^)]+' | \
  xargs -n1 curl -s -o /dev/null -w "%{http_code} %{url_effective}\n"

# The page answers in server HTML, as an AI search crawler sees it
curl -s -A "OAI-SearchBot/1.0" https://your-site.com/pricing | grep -o "<h1[^>]*>[^<]*"

# robots.txt doesn't shut out search crawlers
curl -s https://your-site.com/robots.txt

$ curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://acme.com/llms.txt
200 text/plain; charset=utf-8
$ curl -s https://acme.com/llms.txt | grep -oE 'https?://[^)]+' | xargs -n1 curl ...
200 https://acme.com/docs/quickstart
200 https://acme.com/docs/api
404 https://acme.com/docs/webhooks

• Search crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) are allowed in robots.txt and at the CDN.
• Your important pages have their main content in the server HTML.
• Those pages are indexed in Google and Bing.
• If you publish llms.txt: it returns 200 as plain text, every link resolves, and it is generated from code or reviewed when pages change.
• If you publish developer docs: you also serve llms-full.txt or Markdown copies of each page.

Serpkin practices what this page says. It serves its own /llms.txt from a route handler generated from its page list, so agents can find its docs, and a full llms-full.txt of the docs. Its site scan doesn't flag a missing llms.txt, and its mission for blocked AI crawlers states plainly that an llms.txt file doesn't replace crawler access and isn't a ranking factor. If you want your agent working on the fixes that do matter, npx serpkin connects it and hands it one mission at a time.