SEO for AI-built websites: find and fix what your agent missed

Sites built with Claude Code, Codex, Cursor, Lovable, Bolt or v0 usually fail at SEO for boring reasons, not mysterious ones. The agent builds what you asked for: a page that looks right in a browser. It rarely checks what a crawler receives. The result is often an HTML shell with no text, the same title on every page, no canonical, no sitemap and pages nothing links to. Each of these takes about five minutes to check and, on a modern framework, an afternoon to fix. This guide shows how, then how to brief your agent so it stops happening.

#Why agent-built sites fail at SEO

Coding agents optimize for the request in front of them and for what they can see: the code and, sometimes, a screenshot. Search engines see something else. These are the gaps we find most often:

None of these is specific to AI. They are the classic technical SEO mistakes, made faster. The fix is the same as always: real HTML, one URL per page, unique metadata, a way to be discovered, and content worth indexing.

#Check your site in five minutes

Run these against production, not your local dev server.

1. Look at the raw HTML

   Fetch a page the way a crawler that does not run JavaScript would. If your H1 and main copy are not in the output, you have a rendering problem.

   curl -s https://your-site.com/pricing | grep -o "<h1[^>]*>[^<]*"
   curl -s https://your-site.com/pricing | grep -o "<title>[^<]*"
   curl -s https://your-site.com/pricing | grep -o '<link rel="canonical"[^>]*>'

   In a browser, use view-source (not DevTools, which shows the DOM after JavaScript runs). Repeat for two or three page types: home, a product or pricing page, and a blog post.

2. Compare titles across pages

   for p in / /pricing /about /blog; do
     curl -s "https://your-site.com$p" | grep -o "<title>[^<]*"
   done

   If every line is the same, your metadata is set once in the layout and nowhere else.

3. Check sitemap and robots.txt

   curl -sI https://your-site.com/sitemap.xml | head -1
   curl -s https://your-site.com/robots.txt

   You want a 200 for the sitemap, and a robots.txt that does not say Disallow: / for all user agents (agents sometimes copy that from a staging config).

4. Search with site:

   Search site:your-site.com on Google. Google says the list is not always exhaustive, so use it as a rough signal: zero results on a site that has been live for weeks means something is blocking indexing.

5. Open Search Console

   Verify your domain in Google Search Console, submit the sitemap, and open the Page indexing report. It lists which URLs are indexed and, for the rest, why not. If you see many URLs under "Discovered" or "Crawled – currently not indexed", read our guide to fixing those two statuses.

   Then paste one important URL into the URL Inspection bar and run Test live URL. View tested page shows the HTML Google got after rendering, plus a screenshot. If your copy is missing there, Google can't see it either. The same panel shows the user-declared canonical next to the Google-selected one: if they differ, Google picked another URL as the main version.

#How to fix it, per framework

The biggest fix is rendering: get the content into the HTML the server sends. How you do that depends on the stack your agent chose.

Next.js (App Router)

Next.js renders Server Components on the server by default, so most problems come from moving too much into client components. Fetch data in the page, pass it down as props, and keep "use client" on small interactive pieces.

import type { Metadata } from "next";
import { notFound } from "next/navigation";

export async function generateMetadata({
  params,
}: {
  params: Promise<{ slug: string }>;
}): Promise<Metadata> {
  const { slug } = await params;
  const post = await getPost(slug);
  if (!post) return {};
  return {
    title: post.title,
    description: post.summary,
    alternates: { canonical: `/blog/${slug}` },
  };
}

export default async function PostPage({ params }: { params: Promise<{ slug: string }> }) {
  const { slug } = await params;
  const post = await getPost(slug);
  if (!post) notFound(); // a real 404, not a 200 with "not found" text
  return (
    <article>
      <h1>{post.title}</h1>
      <PostBody html={post.html} />
    </article>
  );
}

One catch: notFound() returns a real 404 status only if it runs before streaming starts, for example when the segment has no loading.tsx. Our Next.js SEO checklist shows how to handle it.

Add app/sitemap.ts, app/robots.ts and metadataBase in the root layout. Our Next.js SEO checklist has the code for each.

Vite + React single-page apps

A plain Vite + React app renders in the browser. The server sends the same near-empty index.html for every route. Google can render JavaScript, but it queues pages for rendering, and Google itself recommends server-side rendering or prerendering because not every bot runs JavaScript. You have three options, from least to most work:

1. Use your platform's prerendering. Lovable's FAQ says older Lovable apps use React + Vite and render in the browser, that it serves prerendered copies of published pages to verified search engines, and that newer apps use TanStack Start, which renders on the server. Check which one you have.
2. Prerender at build time. For marketing pages whose content is known at build time, generate static HTML for each route during the build. This keeps your SPA and fixes the pages that need to rank.
3. Move to a framework with SSR or SSG. Next.js, Astro, Remix/React Router or TanStack Start. If the site is mostly content, this is usually the right long-term call, and coding agents are good at this kind of migration when you give them the route list.

Also make sure every route has its own <title> and canonical in the prerendered HTML, uses real paths (History API, not #/ fragments), and that unknown routes do not return a 200 with an app shell. Google treats that as a soft 404.

Astro and SvelteKit

Astro prerenders pages to static HTML by default and SvelteKit renders on the server by default, so rendering is rarely the problem. Watch for content inside a client-only island (Astro's client:only) or a page with ssr = false (SvelteKit). Both still need a canonical per page, a sitemap and a robots.txt: see the Astro SEO checklist and the SvelteKit SEO checklist.

Everything else

Metadata, canonicals and sitemaps are the same on every stack. Orphan pages and thin content are not framework problems at all. Link every page you care about from at least one related page (see internal linking for small sites), and either improve, merge or noindex templated pages that say nothing new.

#Platform URLs, previews and staging

Builders and hosts give every project extra addresses: your-app.lovable.app, your-project.vercel.app, a netlify.app or pages.dev URL, plus a preview URL per branch. Each one serves the same pages as your real domain. Left alone, Google may index the wrong host or split signals across them.

• Pick one production host (your custom domain, with or without www) and permanently redirect the others to it, path included. Vercel's guide shows a host-scoped redirect for the vercel.app URL; on Lovable, set the custom domain as primary (the lovable.app URL can't be removed).
• Make canonicals absolute and host-fixed. Build them from a production origin constant, not from the request host, or a preview will declare itself canonical.
• Know what your host already does. Vercel adds X-Robots-Tag: noindex to every preview deployment, except when a custom domain is assigned to a non-production branch (a staging. subdomain, for example). Those you must noindex yourself. On Lovable, only publicly published apps can be indexed, and branded workspace URLs never are.
• Don't let staging rules ship. A Disallow: / or a noindex meta tag added for staging is easy for an agent to copy into production. Tie them to an environment variable, not a hard-coded file.
• Read the robots.txt your visitors get, not the one in your repo. Cloudflare's managed robots.txt, when turned on with a restrictive policy, puts Disallow rules for AI crawlers such as GPTBot and ClaudeBot in front of your own rules.

# Previews should say noindex; production should not
curl -sI https://your-branch.vercel.app | grep -i x-robots-tag
curl -sI https://your-site.com | grep -i x-robots-tag

# Platform URL should redirect (301 or 308) to your domain
curl -sI https://your-project.vercel.app/pricing | grep -iE "^(HTTP|location)"

# What robots.txt actually serves in production
curl -s https://your-site.com/robots.txt

#Troubleshooting: symptom, cause, fix

Most problems show up first in Search Console's Page indexing report. The reason names below are the report's own labels.

#How to brief your agent for SEO

"Improve SEO" gets you generic meta tags and maybe a keyword-stuffed paragraph. A useful SEO task for an agent has the same parts as a good engineering ticket: the goal, the evidence, the files, and a way to check it is done.

A prompt that works:

Goal: /pricing should rank for "acme pricing" (Search Console: 320 impressions,
position 11, last 28 days).

Files: app/pricing/page.tsx, components/plans.tsx

Do:
1. Export generateMetadata with a unique title ("Acme pricing: plans for small teams"),
   a 140-160 character description and alternates.canonical "/pricing".
2. Render the FAQ answers on the server (they are inside a client-only component now).

Done when:
- curl -s https://<preview>/pricing shows the new <title>, the canonical and the FAQ text.
- No other routes change.

#An ongoing loop

SEO on an agent-built site is not a one-off audit. Agents change code quickly, and every change can remove a canonical or move content into the browser. Run a small loop instead:

• Every preview deploy keeps titles, canonicals, H1s, robots.txt and the sitemap (curl a few routes).
• Search Console's Page indexing report is checked weekly for new "not indexed" reasons.
• New pages are linked from at least one related page and appear in the sitemap.
• Pages with impressions but low positions get a focused content mission, one at a time.
• You know how your pages show up in AI answers too (see AI Overviews).

Serpkin runs this loop with your agent. It scans your site for issues like content that needs JavaScript to appear, missing canonicals and orphan pages, maps each URL to its source file, and turns each finding into a mission with the files to change and acceptance checks. npx serpkin check <preview-url> compares a preview with production and exits with code 1 on regressions, so it also works in CI. You can start on the free plan with one site.