Semrush

Build me a rank-and-health tracker for sites I own, covering the
self-collectable slice of Semrush. Requirements:

- A Node CLI on a nightly cron: better-sqlite3 for history, and a static HTML
  report regenerated each run.
- Track 20-50 chosen keywords through a SERP API (Serper or DataForSEO, key in
  .env); store position per keyword per day and draw movement sparklines in the
  report.
- Pull clicks, impressions, and average position per query from the Google
  Search Console API for my own properties (OAuth creds in .env).
- A technical crawl of my own site with crawlee: broken links, missing titles
  and descriptions, redirect chains, slow pages, as a table in the report.
- Email the report weekly via Resend (key in .env), but only when something
  moved: a position swing over 3 places or new 404s.
- No accounts, no telemetry; all history lives in one SQLite file.
- Out of scope: competitor keyword gaps, backlink databases, and traffic
  estimates for sites I do not own. Do not scrape or estimate third-party data.
- README: the cheapest SERP API tier for daily checks, the GSC OAuth steps, and
  a note that Semrush's moat is years of crawled keyword, backlink, and SERP
  data no solo build can collect; this tracks only what I own.

# AGENTS.md — Build guide for Semrush

## Project scope
Crawl a site, track selected keywords with an API, collect Google Search Console data, and generate reports.

Catalogue verdict: no. You can build small audits and rank trackers, but Semrush's moat is massive keyword, competitor, backlink, PPC, and SERP datasets plus integrations and reports.
Use the implementation prompt below to define the deliverable. Complete each phase's acceptance checks before extending the scope.

## Working agreement
- Inspect the repository and its existing instructions before choosing paths, dependencies or commands. Keep one coherent stack and explain changes to the proposed architecture.
- Plan a vertical slice that accepts a real input and produces the useful output described below. Persist only the state the prompt calls for; respect memory-only and upstream-managed workflows. Use fixtures only when they are clearly labelled.
- After scaffolding, document the actual install, development, check and build commands in README and keep them synchronized with the package or project manifest. Do not report commands as successful unless they ran.
- Work in small steps. At handoff, list implemented flows, checks actually performed, remaining blockers, and any credentials or provider setup the owner must supply.
- Do not publish, spend money, contact customers, delete source data or run irreversible migrations without the project owner's authorization.

## Prerequisites
- crawler
- search APIs
- GSC/GA integrations
- database
- rank tracking infrastructure
- reporting UI

## Stack and architecture
- A Node CLI on a nightly cron: better-sqlite3 for history, and a static HTML report regenerated each run.
- Model sites, crawl pages, keyword snapshots, audit findings, and report dates; keep stable source IDs and timestamps.
- Interface for this SEO/content/competitive intelligence workflow: a CLI with a dry-run or preview command and an explicit output path.

## Security and data integrity
- When a crawl or search API call fails, retain the last dated snapshot, show its age, and retry within the provider limit.
- Validate imported records and file paths; keep user data within the storage location named in the prompt.
- State the practical limit: keyword database.

## Agent implementation rules
- Project rule, data: Model sites, crawl pages, keyword snapshots, audit findings, and report dates; keep stable source IDs and timestamps.
- Project rule, behavior: Track 20-50 chosen keywords through a SERP API (Serper or DataForSEO, key in .env); store position per keyword per day and draw movement sparklines in the report.
- Project rule, recovery: When a crawl or search API call fails, retain the last dated snapshot, show its age, and retry within the provider limit.

## Optional agent skills and references
No installable skill is listed for this guide; use the project rules and the stack documentation.

Read the linked SKILL.md and its dependencies before adding a skill. Select only the skills matching this project's runtime and task; their documentation does not supply API access, credentials or approval to perform external actions. Pin the reviewed revision where the tool supports it. Follow the chosen agent's documented project-level installation mechanism.

## Distribution ideas
These are optional planning notes. Obtain the owner's approval before publishing or contacting anyone.
- Demonstrate this SEO/content/competitive intelligence operation with sample or user-owned data: Track 20-50 chosen keywords through a SERP API (Serper or DataForSEO, key in .env); store position per keyword per day and draw movement sparklines in the report.
- Show a concrete result or export: A technical crawl of my own site with crawlee: broken links, missing titles and descriptions, redirect chains, slow pages, as a table in the report.
- State the practical limit: keyword database.

## Engineering roadmap
1. Phase 1, architecture and data: A Node CLI on a nightly cron: better-sqlite3 for history, and a static HTML report regenerated each run. Model sites, crawl pages, keyword snapshots, audit findings, and report dates; keep stable source IDs and timestamps.
2. Phase 2, implement: Track 20-50 chosen keywords through a SERP API (Serper or DataForSEO, key in .env); store position per keyword per day and draw movement sparklines in the report.
3. Phase 3, implement: A technical crawl of my own site with crawlee: broken links, missing titles and descriptions, redirect chains, slow pages, as a table in the report.
4. Phase 4, review and output: Crawl a site, track selected keywords with an API, collect Google Search Console data, and generate reports. A technical crawl of my own site with crawlee: broken links, missing titles and descriptions, redirect chains, slow pages, as a table in the report.
5. Phase 5, recovery and acceptance: When a crawl or search API call fails, retain the last dated snapshot, show its age, and retry within the provider limit. Verify this invariant with a saved fixture: A failed crawl retains its last dated snapshot; a report must not imply backlink or keyword coverage the input lacks. State the practical limit: keyword database.

## Paid-product capabilities outside this build
- keyword database
- competitor intelligence
- backlink/PPC data
- AI visibility data
- reports
- integrations
- data freshness

## Implementation prompt
Build me a rank-and-health tracker for sites I own, covering the
self-collectable slice of Semrush. Requirements:

- A Node CLI on a nightly cron: better-sqlite3 for history, and a static HTML
  report regenerated each run.
- Track 20-50 chosen keywords through a SERP API (Serper or DataForSEO, key in
  .env); store position per keyword per day and draw movement sparklines in the
  report.
- Pull clicks, impressions, and average position per query from the Google
  Search Console API for my own properties (OAuth creds in .env).
- A technical crawl of my own site with crawlee: broken links, missing titles
  and descriptions, redirect chains, slow pages, as a table in the report.
- Email the report weekly via Resend (key in .env), but only when something
  moved: a position swing over 3 places or new 404s.
- No accounts, no telemetry; all history lives in one SQLite file.
- Out of scope: competitor keyword gaps, backlink databases, and traffic
  estimates for sites I do not own. Do not scrape or estimate third-party data.
- README: the cheapest SERP API tier for daily checks, the GSC OAuth steps, and
  a note that Semrush's moat is years of crawled keyword, backlink, and SERP
  data no solo build can collect; this tracks only what I own.

## Completion evidence
Demonstrate the prompt's acceptance scenarios against the scoped workflow. Include setup from a clean checkout and failure recovery. Check persistence across restart and export/restore only for the state the prompt says to store; for memory-only tools, confirm that temporary content is discarded as specified. Record actual results and remaining limitations. A detailed plan alone does not establish a working replacement.