MagicChat

Build me a self-hosted documentation chat widget for one site as a limited substitute for MagicChat. Requirements:

- Use Python, FastAPI, SQLite FTS5, and a locally installed Ollama server for embeddings and answers. Store fetched pages and source URLs in one SQLite database.
- Import a sitemap or a user-provided URL list for a site I control, with a page cap and manual refresh. Save title, canonical URL, fetch time, text chunks, and embedding vectors; show pages that could not be fetched.
- Search with FTS5 plus embedding similarity. Retrieve a small set of chunks and ask the local model to answer only from them; return source URLs and quoted excerpts with each answer, or say the docs do not answer it.
- Serve a small embeddable script and chat panel from my server. Allow only configured site origins, rate-limit visitor requests, and keep a visible link to the source documentation.
- Provide an admin view on localhost for import status, failed pages, recent questions, and a manual re-index button. Mask visitor personal data in logs by default.
- No accounts for visitors, telemetry, lead capture, scheduled re-crawls, multi-site tenancy, or claims that a similarity threshold prevents hallucinations. Hosting and model capacity remain my responsibility.
- README: Ollama model setup, public hosting and origin configuration, crawl limits, data path, and how to correct unsupported answers.

EDITORIAL IMPLEMENTATION CONTRACT
Working slice: Index approved documentation, answer visitor questions with cited excerpts and offer a clear 'not found' response when retrieval is insufficient.
Data and invariants: Store SourceURL, CrawlRevision, Chunk, Conversation and Citation; retrieval stays within the one configured site's published knowledge revision and citations reference existing chunks.
Boundary and recovery: Limit crawling to approved hosts and keep admin material out of the public index. An embeddable widget needs origin/rate controls; generated answers still require an escalation path.
Acceptance walkthrough: Ask an unsupported question and include malicious instructions in a document; refuse to invent an answer or follow the document's instructions, while citing valid evidence for supported questions.
Record actual dependency versions, permissions and provider access in setup instructions. Preserve originals, expose partial failures and document backup/restore. These are acceptance requirements, not a claim of a completed or production-certified build. Add the domain, recovery and acceptance rules to AGENTS.md so future edits preserve them.

# AGENTS.md — Build guide for MagicChat

## Project scope
Index approved documentation, answer visitor questions with cited excerpts and offer a clear 'not found' response when retrieval is insufficient.

Catalogue verdict: kinda. A retrieval chatbot over your own docs is one of the most one-shottable products there is: crawl the site, chunk and embed it, answer from the top matches with an LLM, drop in a widget. What you don't get for free is the boring operational layer, scheduled re-crawls, analytics, lead capture and human handoff, multi-source connectors, and a hosted widget that stays up. Buildable in a weekend, real gaps after that.
Use the implementation prompt below to define the deliverable. Complete each phase's acceptance checks before extending the scope.

## Working agreement
- Inspect the repository and its existing instructions before choosing paths, dependencies or commands. Keep one coherent stack and explain changes to the proposed architecture.
- Plan a vertical slice that accepts a real input and produces the useful output described below. Persist only the state the prompt calls for; respect memory-only and upstream-managed workflows. Use fixtures only when they are clearly labelled.
- After scaffolding, document the actual install, development, check and build commands in README and keep them synchronized with the package or project manifest. Do not report commands as successful unless they ran.
- Work in small steps. At handoff, list implemented flows, checks actually performed, remaining blockers, and any credentials or provider setup the owner must supply.
- Do not publish, spend money, contact customers, delete source data or run irreversible migrations without the project owner's authorization.

## Prerequisites
- Runtime and tools: Python, FastAPI, SQLite FTS5 and a locally installed Ollama server for embeddings and answers.
- Before starting: A local Ollama model, Python, an approved sitemap or URL list and explicit public widget origin/rate limits.

## Stack and architecture
- Python, FastAPI, SQLite FTS5 and a locally installed Ollama server for embeddings and answers
- Data design: Store SourceURL, CrawlRevision, Chunk, Conversation and Citation; retrieval stays within the one configured site's published knowledge revision and citations reference existing chunks.
- Setup: A local Ollama model, Python, an approved sitemap or URL list and explicit public widget origin/rate limits

## Security and data integrity
- Keep source evidence, model/config version, draft output and reviewer changes separately. Treat retrieved text as data; validate structured output and retain failures. Never silently send private material to a fallback provider.
- Limit crawling to approved hosts and keep admin material out of the public index. An embeddable widget needs origin/rate controls; generated answers still require an escalation path.
- Keep secrets outside client bundles and exported projects; document what leaves the device and make retention/deletion controls visible.

## Agent implementation rules
- Project rule — domain: Store SourceURL, CrawlRevision, Chunk, Conversation and Citation; retrieval stays within the one configured site's published knowledge revision and citations reference existing chunks.
- Project rule — scope and recovery: Limit crawling to approved hosts and keep admin material out of the public index. An embeddable widget needs origin/rate controls; generated answers still require an escalation path.
- Project rule — acceptance: Ask an unsupported question and include malicious instructions in a document; refuse to invent an answer or follow the document's instructions, while citing valid evidence for supported questions.
- Project rule — delivery: document real setup commands and permissions; do not claim a build, accuracy level, performance result or security certification that has not been demonstrated.

## Optional agent skills and references
- Recommended skill: [modern-python](https://github.com/trailofbits/skills/blob/main/plugins/modern-python/skills/modern-python/SKILL.md) — structure the Python worker or explicitly optional read-only utility with pinned dependencies, typed boundaries and clear failure handling. Follow the maintainer's installation instructions and match its requirements to the chosen runtime.
- Recommended skill: [sharp-edges](https://github.com/trailofbits/skills/blob/main/plugins/sharp-edges/skills/sharp-edges/SKILL.md) — review configuration and API defaults against the app-specific invariants and recovery boundaries above; this is not a security certification. Follow the maintainer's installation instructions and match its requirements to the chosen runtime.
- Recommended skill: [web-design-guidelines](https://github.com/vercel-labs/agent-skills/blob/main/skills/web-design-guidelines/SKILL.md) — review keyboard access, focus, validation, error recovery and the readable work/review interface or HTML report. Follow the maintainer's installation instructions and match its requirements to the chosen runtime.

Read the linked SKILL.md and its dependencies before adding a skill. Select only the skills matching this project's runtime and task; their documentation does not supply API access, credentials or approval to perform external actions. Pin the reviewed revision where the tool supports it. Follow the chosen agent's documented project-level installation mechanism.

## Distribution ideas
These are optional planning notes. Obtain the owner's approval before publishing or contacting anyone.
- Demonstrate this working slice using synthetic or explicitly authorized non-sensitive examples: Index approved documentation, answer visitor questions with cited excerpts and offer a clear 'not found' response when retrieval is insufficient.
- Share a synthetic example export and the acceptance walkthrough; keep real customer, health, financial and source data private: Ask an unsupported question and include malicious instructions in a document; refuse to invent an answer or follow the document's instructions, while citing valid evidence for supported questions.
- State the limits before asking someone to replace their existing tool: Limit crawling to approved hosts and keep admin material out of the public index. An embeddable widget needs origin/rate controls; generated answers still require an escalation path.

## Engineering roadmap
1. Phase 1 — Pin the working slice and create its example input: Index approved documentation, answer visitor questions with cited excerpts and offer a clear 'not found' response when retrieval is insufficient. Confirm setup: A local Ollama model, Python, an approved sitemap or URL list and explicit public widget origin/rate limits.
2. Phase 2 — Implement persistence and write-time invariants before decorating the UI: Store SourceURL, CrawlRevision, Chunk, Conversation and Citation; retrieval stays within the one configured site's published knowledge revision and citations reference existing chunks.
3. Phase 3 — Connect the working view to real saved state. Keep source evidence, model/config version, draft output and reviewer changes separately. Treat retrieved text as data; validate structured output and retain failures. Never silently send private material to a fallback provider.
4. Phase 4 — Expose the app-specific limits and recovery path in context: Limit crawling to approved hosts and keep admin material out of the public index. An embeddable widget needs origin/rate controls; generated answers still require an escalation path.
5. Phase 5 — Walk through this concrete acceptance case and preserve its exported evidence: Ask an unsupported question and include malicious instructions in a document; refuse to invent an answer or follow the document's instructions, while citing valid evidence for supported questions. Finish the README and backup/restore instructions; report unfinished capabilities explicitly.

## Paid-product capabilities outside this build
- scheduled auto re-crawl and content refresh
- analytics and conversation-history dashboards
- lead capture and human handoff
- multi-source connectors and integrations
- hosted uptime for the widget
- team seats and enterprise compliance (HIPAA/DPA/BAA)

## Implementation prompt
Build me a self-hosted documentation chat widget for one site as a limited substitute for MagicChat. Requirements:

- Use Python, FastAPI, SQLite FTS5, and a locally installed Ollama server for embeddings and answers. Store fetched pages and source URLs in one SQLite database.
- Import a sitemap or a user-provided URL list for a site I control, with a page cap and manual refresh. Save title, canonical URL, fetch time, text chunks, and embedding vectors; show pages that could not be fetched.
- Search with FTS5 plus embedding similarity. Retrieve a small set of chunks and ask the local model to answer only from them; return source URLs and quoted excerpts with each answer, or say the docs do not answer it.
- Serve a small embeddable script and chat panel from my server. Allow only configured site origins, rate-limit visitor requests, and keep a visible link to the source documentation.
- Provide an admin view on localhost for import status, failed pages, recent questions, and a manual re-index button. Mask visitor personal data in logs by default.
- No accounts for visitors, telemetry, lead capture, scheduled re-crawls, multi-site tenancy, or claims that a similarity threshold prevents hallucinations. Hosting and model capacity remain my responsibility.
- README: Ollama model setup, public hosting and origin configuration, crawl limits, data path, and how to correct unsupported answers.

EDITORIAL IMPLEMENTATION CONTRACT
Working slice: Index approved documentation, answer visitor questions with cited excerpts and offer a clear 'not found' response when retrieval is insufficient.
Data and invariants: Store SourceURL, CrawlRevision, Chunk, Conversation and Citation; retrieval stays within the one configured site's published knowledge revision and citations reference existing chunks.
Boundary and recovery: Limit crawling to approved hosts and keep admin material out of the public index. An embeddable widget needs origin/rate controls; generated answers still require an escalation path.
Acceptance walkthrough: Ask an unsupported question and include malicious instructions in a document; refuse to invent an answer or follow the document's instructions, while citing valid evidence for supported questions.
Record actual dependency versions, permissions and provider access in setup instructions. Preserve originals, expose partial failures and document backup/restore. These are acceptance requirements, not a claim of a completed or production-certified build. Add the domain, recovery and acceptance rules to AGENTS.md so future edits preserve them.

## Completion evidence
Demonstrate the prompt's acceptance scenarios against the scoped workflow. Include setup from a clean checkout and failure recovery. Check persistence across restart and export/restore only for the state the prompt says to store; for memory-only tools, confirm that temporary content is discarded as specified. Record actual results and remaining limitations. A detailed plan alone does not establish a working replacement.