Afforai

WORKING SLICE
Import permitted PDFs and URLs, inspect extraction, then search the private library using FTS5 and optional local embeddings. Combine ranked results using a documented reciprocal-rank formula and answer only with source-linked passages that open beside the answer.

Build this scoped Afforai-inspired workflow with a documented data model and visible failure states.

Architecture
- Python, FastAPI and server-rendered HTML with HTMX for a local interface.
- SQLite for manifests and job state, with an explicit worker process and immutable source files.
- PyMuPDF extraction with page/block coordinates, SQLite FTS5 and optional local embeddings stored with model IDs.

Prerequisites and limits
A Python virtual environment, writable input/output directories and sufficient disk for both originals and outputs. Bind the service to localhost. Confirm PDF-library licensing and install a local embedding model if semantic retrieval is enabled.
Outside this release: Publisher subscriptions, scholarly citation graphs and factual interpretation are not provided.

Data model and correctness
documents, immutable source hashes, pages, text blocks with coordinates, retrieval chunks, notes and citation spans
Invariant: Every quoted span must match normalized stored text; a source revision invalidates its old embeddings and citations.
Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.

Security and privacy
Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs.

Recovery and export
Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together.

Implementation order
1. Phase 1 — Scope and fixtures. Implement this bounded workflow: Import permitted PDFs and URLs, inspect extraction, then search the private library using FTS5 and optional local embeddings. Combine ranked results using a documented reciprocal-rank formula and answer only with source-linked passages that open beside the answer. Record prerequisites, select representative user-owned fixtures and document the unsupported features: Publisher subscriptions, scholarly citation graphs and factual interpretation are not provided.
2. Phase 2 — Durable model. Model documents, immutable source hashes, pages, text blocks with coordinates, retrieval chunks, notes and citation spans Add migrations or a versioned document format, explicit validation, stable IDs and a visible import-error report. Preserve this rule: Every quoted span must match normalized stored text; a source revision invalidates its old embeddings and citations.
3. Phase 3 — Complete the first useful path. Implement the workflow's input, review and output interface, with clear controls and explicit empty/error states. Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.
4. Phase 4 — Permissions and integration failure. Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs. Request integration credentials and permissions only for the enabled feature; show a disconnected state instead of mock results.
5. Phase 5 — Portable handoff. Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together. Include setup, operating limits, fixture walkthrough and shutdown/restart instructions in the README.
6. Phase 6 — Acceptance scenarios. A question absent from the corpus yields insufficient evidence; a quoted sentence opens the correct PDF page and a scanned PDF shows an OCR-needed state. Repeat the workflow after restart and with a denied permission or unavailable dependency; show recoverable failure rather than a success placeholder.

Acceptance
A question absent from the corpus yields insufficient evidence; a quoted sentence opens the correct PDF page and a scanned PDF shows an OCR-needed state.
Use real source data or clearly labeled fixtures. Explain unsupported input and provider failures; do not fabricate analytics, delivery receipts, accuracy claims or security guarantees.

Optional agent guidance
Optional external skill: [modern-python](https://github.com/trailofbits/skills/blob/main/plugins/modern-python/skills/modern-python/SKILL.md) — Set up Python projects with pyproject.toml, dependency management, linting, typing and automated checks. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Optional external skill: [pdf](https://github.com/anthropics/skills/blob/main/skills/pdf/SKILL.md) — Process PDFs through extraction, generation, page operations, form filling and OCR workflows. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Optional external skill: [sharp-edges](https://github.com/trailofbits/skills/blob/main/plugins/sharp-edges/skills/sharp-edges/SKILL.md) — Review security-sensitive APIs and configuration for dangerous defaults and easy-to-misuse interfaces. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Project rule — data model: documents, immutable source hashes, pages, text blocks with coordinates, retrieval chunks, notes and citation spans
Project rule — preserve this invariant: Every quoted span must match normalized stored text; a source revision invalidates its old embeddings and citations.
Project rule — acceptance evidence: A question absent from the corpus yields insufficient evidence; a quoted sentence opens the correct PDF page and a scanned PDF shows an OCR-needed state.

# AGENTS.md — Build guide for Afforai

## Project scope
Import permitted PDFs and URLs, inspect extraction, then search the private library using FTS5 and optional local embeddings. Combine ranked results using a documented reciprocal-rank formula and answer only with source-linked passages that open beside the answer.

Catalogue verdict: kinda. The visible document research assistant loop is buildable, but a credible replacement needs more than the first screen. Afforai earns its keep through data, import reliability, so expect a weekend or multi-day build and a narrower personal scope.
Use the implementation prompt below to define the deliverable. Complete each phase's acceptance checks before extending the scope.

## Working agreement
- Inspect the repository and its existing instructions before choosing paths, dependencies or commands. Keep one coherent stack and explain changes to the proposed architecture.
- Plan a vertical slice that accepts a real input and produces the useful output described below. Persist only the state the prompt calls for; respect memory-only and upstream-managed workflows. Use fixtures only when they are clearly labelled.
- After scaffolding, document the actual install, development, check and build commands in README and keep them synchronized with the package or project manifest. Do not report commands as successful unless they ran.
- Work in small steps. At handoff, list implemented flows, checks actually performed, remaining blockers, and any credentials or provider setup the owner must supply.
- Do not publish, spend money, contact customers, delete source data or run irreversible migrations without the project owner's authorization.

## Prerequisites
- A Python virtual environment, writable input/output directories and sufficient disk for both originals and outputs. Bind the service to localhost. Confirm PDF-library licensing and install a local embedding model if semantic retrieval is enabled.
- Implementation components: Python, FastAPI and server-rendered HTML with HTMX for a local interface. SQLite for manifests and job state, with an explicit worker process and immutable source files. PyMuPDF extraction with page/block coordinates, SQLite FTS5 and optional local embeddings stored with model IDs.
- Scope boundary: Publisher subscriptions, scholarly citation graphs and factual interpretation are not provided.

## Stack and architecture
- Python, FastAPI and server-rendered HTML with HTMX for a local interface.
- SQLite for manifests and job state, with an explicit worker process and immutable source files.
- PyMuPDF extraction with page/block coordinates, SQLite FTS5 and optional local embeddings stored with model IDs.
- Domain model: documents, immutable source hashes, pages, text blocks with coordinates, retrieval chunks, notes and citation spans

## Security and data integrity
- Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs.
- Correctness boundary: Every quoted span must match normalized stored text; a source revision invalidates its old embeddings and citations.
- Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.
- Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together.

## Agent implementation rules
- Project rule — data model: documents, immutable source hashes, pages, text blocks with coordinates, retrieval chunks, notes and citation spans
- Project rule — preserve this invariant: Every quoted span must match normalized stored text; a source revision invalidates its old embeddings and citations.
- Project rule — acceptance evidence: A question absent from the corpus yields insufficient evidence; a quoted sentence opens the correct PDF page and a scanned PDF shows an OCR-needed state.

## Optional agent skills and references
- Optional external skill: [modern-python](https://github.com/trailofbits/skills/blob/main/plugins/modern-python/skills/modern-python/SKILL.md) — Set up Python projects with pyproject.toml, dependency management, linting, typing and automated checks. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
- Optional external skill: [pdf](https://github.com/anthropics/skills/blob/main/skills/pdf/SKILL.md) — Process PDFs through extraction, generation, page operations, form filling and OCR workflows. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
- Optional external skill: [sharp-edges](https://github.com/trailofbits/skills/blob/main/plugins/sharp-edges/skills/sharp-edges/SKILL.md) — Review security-sensitive APIs and configuration for dangerous defaults and easy-to-misuse interfaces. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.

Read the linked SKILL.md and its dependencies before adding a skill. Select only the skills matching this project's runtime and task; their documentation does not supply API access, credentials or approval to perform external actions. Pin the reviewed revision where the tool supports it. Follow the chosen agent's documented project-level installation mechanism.

## Distribution ideas
These are optional planning notes. Obtain the owner's approval before publishing or contacting anyone.
- Demonstrate the actual Afforai-inspired workflow with owned or clearly labeled sample data: Import permitted PDFs and URLs, inspect extraction, then search the private library using FTS5 and optional local embeddings. Combine ranked results using a documented reciprocal-rank formula and answer only with source-linked passages that open beside the answer.
- Publish a reproducible walkthrough with this observable result: A question absent from the corpus yields insufficient evidence; a quoted sentence opens the correct PDF page and a scanned PDF shows an OCR-needed state.
- Explain who can operate this scoped tool, its setup and ongoing costs, and these remaining product gaps: Publisher subscriptions, scholarly citation graphs and factual interpretation are not provided. Avoid guaranteed savings, performance scores or implied endorsement.

## Engineering roadmap
1. Phase 1 — Scope and fixtures. Implement this bounded workflow: Import permitted PDFs and URLs, inspect extraction, then search the private library using FTS5 and optional local embeddings. Combine ranked results using a documented reciprocal-rank formula and answer only with source-linked passages that open beside the answer. Record prerequisites, select representative user-owned fixtures and document the unsupported features: Publisher subscriptions, scholarly citation graphs and factual interpretation are not provided.
2. Phase 2 — Durable model. Model documents, immutable source hashes, pages, text blocks with coordinates, retrieval chunks, notes and citation spans Add migrations or a versioned document format, explicit validation, stable IDs and a visible import-error report. Preserve this rule: Every quoted span must match normalized stored text; a source revision invalidates its old embeddings and citations.
3. Phase 3 — Complete the first useful path. Implement the workflow's input, review and output interface, with clear controls and explicit empty/error states. Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.
4. Phase 4 — Permissions and integration failure. Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs. Request integration credentials and permissions only for the enabled feature; show a disconnected state instead of mock results.
5. Phase 5 — Portable handoff. Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together. Include setup, operating limits, fixture walkthrough and shutdown/restart instructions in the README.
6. Phase 6 — Acceptance scenarios. A question absent from the corpus yields insufficient evidence; a quoted sentence opens the correct PDF page and a scanned PDF shows an OCR-needed state. Repeat the workflow after restart and with a denied permission or unavailable dependency; show recoverable failure rather than a success placeholder.

## Paid-product capabilities outside this build
- licensed scholarly metadata
- publisher-specific import reliability
- citation graph scale
- team libraries and institutional access

## Implementation prompt
WORKING SLICE
Import permitted PDFs and URLs, inspect extraction, then search the private library using FTS5 and optional local embeddings. Combine ranked results using a documented reciprocal-rank formula and answer only with source-linked passages that open beside the answer.

Build this scoped Afforai-inspired workflow with a documented data model and visible failure states.

Architecture
- Python, FastAPI and server-rendered HTML with HTMX for a local interface.
- SQLite for manifests and job state, with an explicit worker process and immutable source files.
- PyMuPDF extraction with page/block coordinates, SQLite FTS5 and optional local embeddings stored with model IDs.

Prerequisites and limits
A Python virtual environment, writable input/output directories and sufficient disk for both originals and outputs. Bind the service to localhost. Confirm PDF-library licensing and install a local embedding model if semantic retrieval is enabled.
Outside this release: Publisher subscriptions, scholarly citation graphs and factual interpretation are not provided.

Data model and correctness
documents, immutable source hashes, pages, text blocks with coordinates, retrieval chunks, notes and citation spans
Invariant: Every quoted span must match normalized stored text; a source revision invalidates its old embeddings and citations.
Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.

Security and privacy
Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs.

Recovery and export
Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together.

Implementation order
1. Phase 1 — Scope and fixtures. Implement this bounded workflow: Import permitted PDFs and URLs, inspect extraction, then search the private library using FTS5 and optional local embeddings. Combine ranked results using a documented reciprocal-rank formula and answer only with source-linked passages that open beside the answer. Record prerequisites, select representative user-owned fixtures and document the unsupported features: Publisher subscriptions, scholarly citation graphs and factual interpretation are not provided.
2. Phase 2 — Durable model. Model documents, immutable source hashes, pages, text blocks with coordinates, retrieval chunks, notes and citation spans Add migrations or a versioned document format, explicit validation, stable IDs and a visible import-error report. Preserve this rule: Every quoted span must match normalized stored text; a source revision invalidates its old embeddings and citations.
3. Phase 3 — Complete the first useful path. Implement the workflow's input, review and output interface, with clear controls and explicit empty/error states. Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.
4. Phase 4 — Permissions and integration failure. Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs. Request integration credentials and permissions only for the enabled feature; show a disconnected state instead of mock results.
5. Phase 5 — Portable handoff. Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together. Include setup, operating limits, fixture walkthrough and shutdown/restart instructions in the README.
6. Phase 6 — Acceptance scenarios. A question absent from the corpus yields insufficient evidence; a quoted sentence opens the correct PDF page and a scanned PDF shows an OCR-needed state. Repeat the workflow after restart and with a denied permission or unavailable dependency; show recoverable failure rather than a success placeholder.

Acceptance
A question absent from the corpus yields insufficient evidence; a quoted sentence opens the correct PDF page and a scanned PDF shows an OCR-needed state.
Use real source data or clearly labeled fixtures. Explain unsupported input and provider failures; do not fabricate analytics, delivery receipts, accuracy claims or security guarantees.

Optional agent guidance
Optional external skill: [modern-python](https://github.com/trailofbits/skills/blob/main/plugins/modern-python/skills/modern-python/SKILL.md) — Set up Python projects with pyproject.toml, dependency management, linting, typing and automated checks. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Optional external skill: [pdf](https://github.com/anthropics/skills/blob/main/skills/pdf/SKILL.md) — Process PDFs through extraction, generation, page operations, form filling and OCR workflows. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Optional external skill: [sharp-edges](https://github.com/trailofbits/skills/blob/main/plugins/sharp-edges/skills/sharp-edges/SKILL.md) — Review security-sensitive APIs and configuration for dangerous defaults and easy-to-misuse interfaces. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Project rule — data model: documents, immutable source hashes, pages, text blocks with coordinates, retrieval chunks, notes and citation spans
Project rule — preserve this invariant: Every quoted span must match normalized stored text; a source revision invalidates its old embeddings and citations.
Project rule — acceptance evidence: A question absent from the corpus yields insufficient evidence; a quoted sentence opens the correct PDF page and a scanned PDF shows an OCR-needed state.

## Completion evidence
Demonstrate the prompt's acceptance scenarios against the scoped workflow. Include setup from a clean checkout and failure recovery. Check persistence across restart and export/restore only for the state the prompt says to store; for memory-only tools, confirm that temporary content is discarded as specified. Record actual results and remaining limitations. A detailed plan alone does not establish a working replacement.