In short
- Competitor watch reads public sitemaps weekly, with no credits and no search data, and respects robots.txt.
- Free checks 2 competitors per site and paid plans check 5; you can add up to 5 per site by hand.
- The first check is a baseline; later checks list pages that weren't in the sitemap before, minus tags, feeds and other noise.
- A clear topic your site doesn't cover can become a mission, at most 2 per refresh.
Competitor watch reads your competitors' public sitemaps every week and lists the pages they publish. When a competitor covers a topic your site doesn't, it can become a mission for your agent. It uses no credits and no search data, only what each competitor already publishes for search engines, and it respects their robots.txt.
Add a competitor
- Open Results, then the Competitors tab
Each site has its own list of competitors.
- Enter a domain, such as rival.com, and choose Watch
You can add up to 5 competitors per site by hand. Competitors found by keyword research appear in the list too; the ones you add come first.
- Wait for the first check
The first check is queued within the hour. After that, each site is checked once a week, at a fixed time between Monday and Wednesday (UTC).
| Plan | Competitors checked each week |
|---|---|
| Free | 2 |
| Builder ($29 a month) | 5 |
| Pro ($79 a month) | 5 |
On Free, competitors beyond the first two show as "Not checked on your plan". Free workspaces with no sign-in or agent activity for 30 days are paused until you're back. Removing a competitor asks for confirmation, because it deletes its baseline.
How it reads a competitor
Serpkin finds the sitemap from the competitor's robots.txt, or at /sitemap.xml, and reads it the way a search engine does, within bounds: up to 6 sitemap files and 5,000 URLs per competitor. For up to 10 new pages each week it fetches the page once to read its title and heading.
What counts as new
- 1First checkRecords every URL as the baseline. Nothing is called new yet.
- 2Each weekCompares the sitemap with last week's and finds URLs that weren't there.
- 3FilteredDrops pages that aren't real content, and duplicates of the same page.
- 4LabeledReads the title and sorts the page: guide, comparison, tool, blog post and more.
On the first check you may still see a few pages marked Recently updated. Those come from sitemap dates within the last 30 days, and they mean the page changed recently, not that it's new. Sitemaps that stamp every URL with the same build time are ignored, because those dates say nothing about publishing.
Shown
- Good: A page that wasn't in last week's sitemap
- Good: Guides, comparisons, free tools, blog posts, docs and landing pages
- Good: One entry per page, even when two URLs show the same page
Ignored
- Problem: Pagination, tag, author and feed pages, and assets
- Problem: URLs with query strings and other-language copies
- Problem: Pages marked noindex, and pages that return an error
If more than 50 URLs, or 30% of a sitemap, change in one week, that usually means a redesign or migration rather than publishing. Serpkin treats that week as a new baseline instead of flooding your feed.
From a competitor page to a mission
A competitor publishing on a topic is a signal, not proof that people search for it. So Serpkin only suggests a mission when the topic is clear and your site has no page on it:
| Competitor page | Mission |
|---|---|
| Comparison or alternatives page | A comparison page for your site |
| Free tool or calculator | A free tool mission |
| Guide or blog post | A new page on the topic |
| Landing page, docs, changelog, integrations or customer story | No mission: these describe the competitor's own product |
The rules
- Only pages first seen in the last 45 days, and never "Recently updated" baseline items.
- A topic counts as covered when one of your pages shares at least half its meaningful words. Vague topics count as covered, so missions only appear for clear gaps.
- At most 2 new missions per refresh, with a modest priority, so they never crowd out fixes or near-page-one wins.
- Each mission links to the competitor page as its evidence. Write your own, more useful page; don't copy theirs.
Where you see it
The Competitors tab on Results shows the 20 most recent pages, grouped by competitor, each with its type, its path and when it was first seen, plus a link to any mission it created. The weekly email adds one line per competitor, such as "rival.com published 3 new pages", and your agent sees the same in serpkin_weekly_summary. For the missions themselves, see Missions.
Questions
Why does my feed say the first check set a baseline?
The first check can't tell old pages from new ones, so it records every URL. New pages show up from the following week, apart from a few marked Recently updated from recent sitemap dates.
Does Serpkin crawl my competitors' whole sites?
No. It reads their sitemaps, within bounds, and fetches up to 10 new pages per competitor each week to read their titles. It never fetches pages that robots.txt disallows.