Oncrawl

Build me a focused technical audits and on-page optimization workflow for the personal core of Oncrawl. Requirements:

- Use Python 3.12, FastAPI, SQLite, Playwright, and an HTMX interface. Model sites, crawl pages, keyword snapshots, audit findings, and report dates; keep stable source IDs and timestamps.
- Paid product context: Crawl one owned site locally and join pages to user-supplied analytics exports. Build only this DIY scope: Crawl one owned site locally, inspect HTML and rendered pages, join pages to user-supplied analytics exports, explain prioritized issues, and export a reproducible audit.
- Collect status, title, description, headings, canonical, robots, links, images, structured data, and rendered text.
- Detect duplicates, orphan candidates, broken links, redirect chains, missing metadata, and indexability conflicts.
- Show every issue with affected URLs, evidence, severity, and a concrete remediation note. Export crawl data and issues to CSV plus a self-contained HTML report.
- Use a local web page with input, progress, review, and export views. Required input or access: Playwright browsers; permission to crawl the target site.
- Recovery: Require an explicit ownership or permission acknowledgement before a crawl starts.
- Acceptance: with one labelled sample, show the input, saved intermediate state, and exported result; verify this invariant: A failed crawl retains its last dated snapshot; a report must not imply backlink or keyword coverage the input lacks.
- Out of scope: large-scale crawler, log analysis, data connectors, and enterprise support; massive hosted crawl capacity. Keep this a personal, inspectable workflow.
- Include a README with setup, a sample input, required keys or permissions, data location, and the supported scope.

# AGENTS.md — Build guide for Oncrawl

## Project scope
Crawl one owned site locally, inspect HTML and rendered pages, join pages to user-supplied analytics exports, explain prioritized issues, and export a reproducible audit.

Catalogue verdict: no. A consolation build is possible, but the paid product's decisive value sits outside a solo rebuild. For Oncrawl, crawl one owned site locally and join pages to user-supplied analytics exports. The hard boundary is large-scale crawler, log analysis, data connectors, and enterprise support, plus crawl scale, rule depth, and operational polish.
Use the implementation prompt below to define the deliverable. Complete each phase's acceptance checks before extending the scope.

## Working agreement
- Inspect the repository and its existing instructions before choosing paths, dependencies or commands. Keep one coherent stack and explain changes to the proposed architecture.
- Plan a vertical slice that accepts a real input and produces the useful output described below. Persist only the state the prompt calls for; respect memory-only and upstream-managed workflows. Use fixtures only when they are clearly labelled.
- After scaffolding, document the actual install, development, check and build commands in README and keep them synchronized with the package or project manifest. Do not report commands as successful unless they ran.
- Work in small steps. At handoff, list implemented flows, checks actually performed, remaining blockers, and any credentials or provider setup the owner must supply.
- Do not publish, spend money, contact customers, delete source data or run irreversible migrations without the project owner's authorization.

## Prerequisites
- Python 3.12
- Playwright browsers
- permission to crawl the target site
- local disk space

## Stack and architecture
- Use Python 3.12, FastAPI, SQLite, Playwright, and an HTMX interface.
- Model sites, crawl pages, keyword snapshots, audit findings, and report dates; keep stable source IDs and timestamps.
- Interface for this technical audits and on-page optimization workflow: a local web page with input, progress, review, and export views.

## Security and data integrity
- Require an explicit ownership or permission acknowledgement before a crawl starts.
- Validate imported records and file paths; keep user data within the storage location named in the prompt.
- State the practical limit: large-scale crawler, log analysis, data connectors, and enterprise support.

## Agent implementation rules
- Project rule, data: Model sites, crawl pages, keyword snapshots, audit findings, and report dates; keep stable source IDs and timestamps.
- Project rule, behavior: Collect status, title, description, headings, canonical, robots, links, images, structured data, and rendered text.
- Project rule, recovery: Require an explicit ownership or permission acknowledgement before a crawl starts.

## Optional agent skills and references
No installable skill is listed for this guide; use the project rules and the stack documentation.

Read the linked SKILL.md and its dependencies before adding a skill. Select only the skills matching this project's runtime and task; their documentation does not supply API access, credentials or approval to perform external actions. Pin the reviewed revision where the tool supports it. Follow the chosen agent's documented project-level installation mechanism.

## Distribution ideas
These are optional planning notes. Obtain the owner's approval before publishing or contacting anyone.
- Demonstrate this technical audits and on-page optimization operation with sample or user-owned data: Collect status, title, description, headings, canonical, robots, links, images, structured data, and rendered text.
- Show a concrete result or export: Export crawl data and issues to CSV plus a self-contained HTML report.
- State the practical limit: large-scale crawler, log analysis, data connectors, and enterprise support.

## Engineering roadmap
1. Phase 1, architecture and data: Use Python 3.12, FastAPI, SQLite, Playwright, and an HTMX interface. Model sites, crawl pages, keyword snapshots, audit findings, and report dates; keep stable source IDs and timestamps.
2. Phase 2, implement: Collect status, title, description, headings, canonical, robots, links, images, structured data, and rendered text.
3. Phase 3, implement: Detect duplicates, orphan candidates, broken links, redirect chains, missing metadata, and indexability conflicts.
4. Phase 4, review and output: Show every issue with affected URLs, evidence, severity, and a concrete remediation note. Export crawl data and issues to CSV plus a self-contained HTML report.
5. Phase 5, recovery and acceptance: Require an explicit ownership or permission acknowledgement before a crawl starts. Verify this invariant with a saved fixture: A failed crawl retains its last dated snapshot; a report must not imply backlink or keyword coverage the input lacks. State the practical limit: large-scale crawler, log analysis, data connectors, and enterprise support.

## Paid-product capabilities outside this build
- large-scale crawler, log analysis, data connectors, and enterprise support
- massive hosted crawl capacity
- proprietary scoring
- continuous monitoring
- agency reporting and support

## Implementation prompt
Build me a focused technical audits and on-page optimization workflow for the personal core of Oncrawl. Requirements:

- Use Python 3.12, FastAPI, SQLite, Playwright, and an HTMX interface. Model sites, crawl pages, keyword snapshots, audit findings, and report dates; keep stable source IDs and timestamps.
- Paid product context: Crawl one owned site locally and join pages to user-supplied analytics exports. Build only this DIY scope: Crawl one owned site locally, inspect HTML and rendered pages, join pages to user-supplied analytics exports, explain prioritized issues, and export a reproducible audit.
- Collect status, title, description, headings, canonical, robots, links, images, structured data, and rendered text.
- Detect duplicates, orphan candidates, broken links, redirect chains, missing metadata, and indexability conflicts.
- Show every issue with affected URLs, evidence, severity, and a concrete remediation note. Export crawl data and issues to CSV plus a self-contained HTML report.
- Use a local web page with input, progress, review, and export views. Required input or access: Playwright browsers; permission to crawl the target site.
- Recovery: Require an explicit ownership or permission acknowledgement before a crawl starts.
- Acceptance: with one labelled sample, show the input, saved intermediate state, and exported result; verify this invariant: A failed crawl retains its last dated snapshot; a report must not imply backlink or keyword coverage the input lacks.
- Out of scope: large-scale crawler, log analysis, data connectors, and enterprise support; massive hosted crawl capacity. Keep this a personal, inspectable workflow.
- Include a README with setup, a sample input, required keys or permissions, data location, and the supported scope.

## Completion evidence
Demonstrate the prompt's acceptance scenarios against the scoped workflow. Include setup from a clean checkout and failure recovery. Check persistence across restart and export/restore only for the state the prompt says to store; for memory-only tools, confirm that temporary content is discarded as specified. Record actual results and remaining limitations. A detailed plan alone does not establish a working replacement.