Stop your coding agent from breaking SEO

Coding agents break SEO for one reason: they optimize for the task in front of them, and the things search engines depend on (a canonical tag, a title that ranks, a URL that has backlinks) are invisible in that task. The diff looks clean, the tests pass, and the page quietly loses its canonical or its server-rendered content. You stop it with three layers: rules the agent reads at the start of every session, a check of the preview's real server HTML before merge, and a check of production after deploy. This guide gives you all three for a Next.js App Router project, with notes for Vite and Astro.

#How agents break SEO

None of these are exotic. They are normal refactors that happen to touch something search engines read. The agent can't see your Search Console data, doesn't know which page earns your signups, and starts each session with no memory of last week's fix unless you write it down.

The two that do the most damage

Content leaving the server HTML. An agent asked to add a filter or a loading spinner often converts the whole page to a client component. The browser shows the same thing a second later, so the agent reports success.

Google does render JavaScript, but it queues pages for rendering and says server-side or pre-rendering is still a great idea because not all bots can run JavaScript. Many AI crawlers read raw HTML only. A "use client" file also can't export metadata or generateMetadata, so an agent that moves the directive to the page file has to move or drop the metadata too.

Metadata lost in a merge. Next.js merges layout and page metadata one level deep. If a page sets alternates: { languages }, it replaces the whole alternates object, including canonical. An agent adding hreflang to a page that relied on a layout canonical removes it, and nothing on screen changes. The fix is a helper that always sets both (below).

#Write the rules where the agent reads them

Each agent loads a project instruction file when a session starts. Codex reads AGENTS.md before doing any work. Cursor reads AGENTS.md and .mdc rules in .cursor/rules. Claude Code reads CLAUDE.md, and reads AGENTS.md when there is no CLAUDE.md; if you have both, import one from the other with a line containing @AGENTS.md. Write the SEO rules once in AGENTS.md and point the others at it.

## SEO rules (read before editing pages, layouts, routes or config)

- Pages under app/ stay Server Components. Fetch page content on the server.
  Put "use client" only on small interactive leaves; never on a page.tsx or layout.tsx.
- Every indexable page exports metadata or generateMetadata built with pageMeta()
  from lib/seo.ts. Never set alternates, robots or openGraph by hand.
- Do not change the title, description or H1 of any path in seo/protected.json
  without asking me first. Propose the change and wait.
- Do not delete, rename or move a route without adding a permanent redirect in
  next.config.ts, and tell me the old and new URL.
- Never add noindex or a Disallow rule outside the preview branch of app/robots.ts.
- Keep app/sitemap.ts listing every path in seo/protected.json.
- Before saying a page change is done, run:
  node scripts/seo-check.mjs http://localhost:3000
  against a production build (pnpm build && pnpm start) and paste the output.

---
description: SEO rules for pages, layouts, routes and config
globs: app/**,next.config.ts,lib/seo.ts,seo/**
alwaysApply: false
---
Follow the "SEO rules" section in AGENTS.md. Run scripts/seo-check.mjs before
reporting a page change as done.

Keep the rules short and concrete. Claude Code's docs say instruction files are context, not enforced configuration, and that specific, concise rules are followed more consistently. That is why every rule above points at a file or a command: the agent can check itself, and you can check the agent. If you need a hard block (for example, no edits to seo/protected.json), Claude Code supports a PreToolUse hook for that.

#Keep SEO values in one module

Agents copy patterns. If titles, canonicals and robots rules are spread across forty pages, each one is a chance to drop something. Put them in one module so a page can only get them right:

import type { Metadata } from "next";

export const SITE_URL = "https://your-site.com"; // production origin, never VERCEL_URL

type PageMetaInput = {
  path: string; // "/pricing"
  title: string;
  description: string;
  languages?: Record<string, string>;
  noindex?: boolean; // only for thin utility pages, never for protected ones
};

/** The only way pages should build metadata: canonical and robots are always set. */
export function pageMeta({ path, title, description, languages, noindex }: PageMetaInput): Metadata {
  return {
    title,
    description,
    alternates: { canonical: path, ...(languages ? { languages } : {}) },
    robots: noindex ? { index: false, follow: true } : { index: true, follow: true },
    openGraph: { title, description, url: path, siteName: "Acme", images: ["/og/default.png"] },
  };
}

import { pageMeta } from "@/lib/seo";

export const metadata = pageMeta({
  path: "/pricing",
  title: "Pricing",
  description: "Plans for teams of 1 to 50, billed monthly or yearly.",
});

Set metadataBase: new URL(SITE_URL) in the root layout so relative canonicals become absolute. The Next.js SEO checklist covers the rest of the setup.

#Protect the pages that earn traffic

An agent can't tell a dead page from one that brings in half your signups. Give it a list. In Search Console, open Performance, sort pages by clicks for the last three months, and copy the top ones into a file the rules and the check both read:

[
  "/",
  "/pricing",
  "/blog/nextjs-auth-guide",
  "/tools/invoice-generator",
  "/alternatives/acme-vs-other"
]

Refresh it every month or so. Pages on this list keep their URL, return 200, stay in the sitemap and don't get new titles or H1s without a human saying yes. When a protected URL really must move, map it to its closest replacement with a permanent redirect. Google recommends server-side 301 or 308 redirects, kept for at least a year, pointing straight at the final URL rather than at the homepage:

import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  async redirects() {
    return [
      // Renamed 2026-11: keep for at least a year.
      { source: "/pricing", destination: "/plans", permanent: true }, // 308
    ];
  },
};
export default nextConfig;

#Check the preview's server HTML before merge

Review the result, not the diff. A diff tells you what the agent changed; the server response tells you what Google will get. These four lines cover most regressions in a few seconds:

URL=https://your-branch.vercel.app/pricing
curl -s "$URL" | grep -oE "<title>[^<]*</title>"
curl -s "$URL" | grep -oE '<link rel="canonical"[^>]*>'
curl -s "$URL" | grep -oiE '<meta name="robots"[^>]*>'
curl -s "$URL" | grep -oE "<h1[^>]*>[^<]*" | head -1

You want a real title, a canonical pointing at the production URL, no noindex in the robots meta tag, and your H1 text. If the H1 is missing but shows in the browser, the content moved to the client. Run the same lines against production and compare. On Vercel, preview deployments send an X-Robots-Tag: noindex header by default, so a header noindex on the preview is expected; a noindex in the page's own meta tag is your code and will ship. For a quick one-off read of any URL, the free SEO checker shows the same fields.

Check renamed URLs too:

curl -sI https://your-branch.vercel.app/old-path | grep -iE "^(HTTP|location)"   # want 308 + new URL

#Fail CI on noindex, missing canonicals and sitemap gaps

Manual curl works until you forget it. The same checks fit in a short Node script with no dependencies (Node 22 has fetch built in). It reads seo/protected.json, so the agent's rules and the check never drift apart:

// node scripts/seo-check.mjs <base-url>   (SEO_CHECK_PROD=1 also fails on header noindex)
import { readFile } from "node:fs/promises";

const SITE_URL = "https://your-site.com";
const base = process.argv[2]?.replace(/\/$/, "");
if (!base) { console.error("usage: seo-check.mjs <base-url>"); process.exit(2); }
const prod = process.env.SEO_CHECK_PROD === "1";
const paths = JSON.parse(await readFile("seo/protected.json", "utf8"));
const errors = [];
const fail = (path, msg) => errors.push(`${path}: ${msg}`);

for (const path of paths) {
  const res = await fetch(base + path, { redirect: "manual" });
  if ([301, 308].includes(res.status)) continue; // moved on purpose
  if (res.status !== 200) { fail(path, `status ${res.status}`); continue; }
  const html = await res.text();
  if (!/<title>[^<]+<\/title>/i.test(html)) fail(path, "no title");
  if (!/<link[^>]+rel="canonical"[^>]+href="https?:\/\//i.test(html)) fail(path, "no absolute canonical");
  if (/<meta[^>]+name="robots"[^>]+noindex/i.test(html)) fail(path, "noindex in meta robots");
  if (prod && /noindex/i.test(res.headers.get("x-robots-tag") ?? "")) fail(path, "X-Robots-Tag noindex");
  if (!/<h1[\s>][\s\S]*?<\/h1>/i.test(html)) fail(path, "no H1 in server HTML");
}

const sitemap = await fetch(base + "/sitemap.xml");
const xml = sitemap.ok ? await sitemap.text() : "";
if (!xml.includes("<urlset") && !xml.includes("<sitemapindex")) fail("/sitemap.xml", `not a sitemap (${sitemap.status})`);
const listed = (p) => [SITE_URL + p, SITE_URL + p.replace(/\/$/, "")].some((u) => xml.includes(`<loc>${u}</loc>`));
if (xml) for (const path of paths) if (!listed(path)) fail(path, "missing from sitemap");

if (prod) {
  const robots = await (await fetch(base + "/robots.txt")).text(); // simple: flags any group, so adjust if you block specific bots
  if (/^Disallow:\s*\/\s*$/im.test(robots)) fail("/robots.txt", "Disallow: / on production");
}

console.log(errors.length ? errors.join("\n") : `ok: ${paths.length} protected paths`);
process.exit(errors.length ? 1 : 0);

The sitemap check compares against SITE_URL, because a correct sitemap lists production URLs even when served from a preview. If your sitemap uses a different trailing-slash style, match it here. Run it on every preview and on a schedule against production:

name: SEO check
on:
  deployment_status:
  schedule:
    - cron: "0 7 * * *" # daily production check
  workflow_dispatch:

jobs:
  preview:
    if: github.event_name == 'deployment_status' && github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: node scripts/seo-check.mjs "${{ github.event.deployment_status.target_url }}"

  production:
    if: github.event_name != 'deployment_status'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: node scripts/seo-check.mjs https://your-site.com
        env: { SEO_CHECK_PROD: "1" }

Vercel sends deployment_status events when a preview finishes. If your previews use Deployment Protection, the script gets a 401; allow automated requests through or the check fails on every path. Make the job a required status check so an agent-written pull request can't merge while it is red.

#Vite and Astro

The failure modes are the same; where they hide differs.

• Vite (React SPA). Everything is client-rendered unless you prerender, so the server HTML is your index.html shell. The common agent regression is moving titles into a client-side head library and calling it fixed. Run the script against the built output served by vite preview: if every page shows the same title and no H1, nothing is prerendered. See React and Vite SEO.
• Astro. Pages are static HTML by default, which is the safe state. Watch for client:only on a component that holds main content: Astro skips server rendering for it. Redirects in astro.config default to 301, but in static output without an adapter Astro writes meta refresh pages instead of HTTP redirects. @astrojs/sitemap needs site set in the config. See Astro SEO.

In both, write the same rules into AGENTS.md with the file names changed, and keep seo/protected.json and the check script as they are: they only speak HTTP.

#Verify the live result after deploy

1. Re-run the check on production

   A passing preview isn't proof. Environment variables differ, a robots.ts environment check can fail in production, and caches can serve the old page. Run SEO_CHECK_PROD=1 node scripts/seo-check.mjs https://your-site.com after each production deploy.

2. Inspect changed pages in Search Console

   Use the URL Inspection live test on pages the change touched. It fetches the page in real time and shows the HTML and a screenshot as Google sees them. The indexed result is Google's stored copy, not the live page. If you fixed something, request indexing; Google says it usually takes about a day but isn't guaranteed.

3. Watch protected pages for two to four weeks

   In Performance, filter by page and compare clicks and impressions with the period before the deploy. Check the Page indexing report for new "Excluded by noindex tag" or redirect entries. A noindex removes a page when Google next crawls it, so an accidental one can show up days after the deploy that caused it.

If a protected page drops, diff its server HTML now against what it served before the change, not just the code. That is usually where the cause is.

#Where Serpkin fits

Serpkin automates the preview-versus-production comparison and the after-deploy checks. npx serpkin check <preview-url> compares the homepage plus up to 10 known pages (up to 25 paths) on the preview and on production. It fails on HTTP errors, a noindex added in the robots meta tag, a removed title, description or H1, a canonical that is removed, invalid or points at another site, and a broken robots.txt or sitemap. A renamed route without a redirect shows up as an HTTP error on the old path. Problems production already has are warnings, and Vercel's preview noindex header is never counted as a regression. It exits with code 1, so it drops into the workflow above (npx serpkin weekly writes a ready-made one; see GitHub Actions). It flags removed titles, not rewritten ones, so the protected-pages rule still matters.

Missions your agent implements are verified on the preview and again on production. With a deploy hook, a private URL you call after each production deploy, Serpkin re-checks waiting missions on the live site and runs a site watch check right away. Site watch also re-checks verified sites daily on Builder and Pro and alerts you when a page gains noindex, its canonical changes, the sitemap breaks or robots.txt starts blocking pages. Builder and Pro start with a 7-day free trial.

#Checklist

• SEO rules live in AGENTS.md (and CLAUDE.md imports it, if you have one).
• Every page builds metadata through one helper that always sets canonical and robots.
• seo/protected.json lists the pages that earn clicks, refreshed monthly.
• Renamed or removed protected URLs get a 308 to their closest replacement.
• The preview's server HTML has title, canonical, H1 and no meta noindex before merge.
• CI runs the check on every preview and daily on production, and it is a required status.
• After each deploy you check production, then watch Search Console for the protected pages.

For running SEO work through agents beyond guarding against regressions, see managing SEO with AI agents.