Ingesting documents

remember is for one statement. ingest is for everything longer than that.

sigil ingest ./DESIGN.md
sigil ingest "docs/**/*.md"
sigil ingest https://example.com/rfc-0042 --wait
sigil ingest notes-a.md notes-b.md --namespace=engineering

The important property: ingest keeps the full original text and extracts facts from it. You get both layers — searchable atoms for recall, and the source document when the atoms aren’t enough.

What gets stored

1. Admission. The document is hashed for change detection. Unchanged content is skipped early rather than re-embedded. The full text is persisted on the document row so it can be handed back whole later.

2. Secret masking. Recognized secret patterns are masked at this single choke point — before classification, chunking, embedding, and extraction. Every downstream copy is masked, so a credential never reaches your embedding provider or lands in the database.

3. Parse. Content is parsed into text plus section structure.

4. Chunk. Text is split into ~512-token blocks with overlap at the seams.

5. Contextualize. Each chunk gets a short generated preamble situating it in the whole document. A chunk that says “this is capped at 30 days” is useless in isolation; contextualized, it says what this is.

6. Embed. Each chunk is embedded with your configured provider, at 1024 dimensions.

7. Extract. Atomic facts are pulled from the chunks, each with a category, a confidence, an importance, and a link back to its source section.

8. Link. Entities are extracted and connected into the graph. This runs as a durable background job by default rather than blocking the write.

Everything is attached to the pod for the directory you ingested from, so a spec ingested inside one repo doesn’t bleed into another.

Sources

Source	Example
A file	sigil ingest ./ARCHITECTURE.md
Several files	sigil ingest a.md b.md c.md
A glob	sigil ingest "docs/**/*.md" — quote it, or your shell expands it first
A URL	sigil ingest https://internal.wiki/oncall

Options

Flag	Effect
--wait	Block until ingestion finishes and report results. Default is to queue in the background and return immediately.
--namespace=<ns>	Target namespace
--skip-facts	Store and embed chunks, but extract no facts
--skip-entities	Skip entity linking

Background is the default because a large document is minutes of LLM work. Watch progress on the dashboard’s Engine screen, or:

sigil status

Use --wait in scripts

In CI or a setup script, --wait is what you want — otherwise the process exits before the queue drains and you can’t tell success from silence.

Reading documents back

sigil docs                       # documents for the current project
sigil docs --all                 # every project
sigil docs doc-a1b2c3            # print one in full
sigil docs --type=url --limit=10

Search finds the facts; docs gets you the source. When an injected fact came from a document, its source is shown — fetch the document only when the facts alone don’t answer the question. Pulling a whole document “just in case” is what the fact layer exists to avoid.

MCP clients get the same two operations as list_documents and get_document.

What to ingest

Worth it:

• Architecture docs, design docs, ADRs
• Runbooks and postmortems
• RFCs and specs, internal or external
• Meeting transcripts where decisions were made
• API documentation for services you integrate with

Not worth it:

• Source code — your agent reads the repo directly, and code chunks embed poorly
• Anything auto-generated; it churns and re-ingests forever
• Files that change hourly

Re-ingesting

Re-ingesting the same source is safe. The content hash short-circuits unchanged documents, and when content has changed, the new version supersedes the old one rather than duplicating it — facts from the previous version are marked superseded, not deleted, so provenance survives.

sigil ingest ./DESIGN.md    # after editing it — updates in place

Tuning

Three settings control the cost/depth trade-off, all in Settings → Ingestion:

Setting	Default	What it does
ingest.eagerExtract	true	Extract facts at write time. Turn off to store chunks only and extract later.
ingest.extractRelations	false	Discover graph relations during ingest. Off by default — a coalesced background job handles cross-document edges more cheaply.
ingest.graphGleanRounds	0	Extra passes over each chunk hunting for missed entities. Each round costs another LLM call per chunk.
ingest.workerConcurrency	2	Background workers. Raise for throughput on a fast provider; needs a daemon restart.

Full list in Configuration.

Cost

Ingest is the expensive operation in Sigil — one embedding call per chunk plus extraction calls per chunk. A 5,000-word document is on the order of twenty chunks.

To make it free: use Ollama for both LLM and embeddings. Slower, entirely local, no per-token cost. See Air-gapped setup.