Hybrid search

Neither vector search nor keyword search is sufficient alone. Sigil runs both and fuses them — in a single SQL query, which is why search is 33ms and not a multi-round-trip pipeline.

Why both

Vector search matches meaning. “How do we deliver events” finds “Postgres LISTEN/NOTIFY for event delivery” with no shared vocabulary at all. But it blurs exact tokens: search for SKIP LOCKED and the embedding cheerfully returns things about locking in general.

Keyword search matches tokens. It nails identifiers, error codes, function names, version numbers — the things you’d grep for. But it returns nothing when you phrase the same idea differently.

Real queries need both. “Why did the payment_id idempotency change” is half concept, half identifier.

Reciprocal Rank Fusion

The two searches produce two ranked lists whose scores aren’t comparable — cosine similarity and BM25 don’t live on the same scale, and normalizing them is fragile.

RRF sidesteps that by ignoring scores and using only rank:

score(fact) = Σ  weight_i / (K + rank_i(fact))

Constant	Value	Why
K	20	The original paper’s K=60 compresses scores into a ~0.001 band at our result-set sizes (5–50). K=20 gives usable spread.
Vector weight	1.0	Better for semantic and natural-language queries, which is most of them
Keyword weight	0.7	Precise when it hits, but narrower

A fact ranked #1 by vector and #3 by keyword beats one ranked #2 by both. Rank-based fusion is robust to either search having a weird score distribution on a given query.

The pipeline

query
  │
  ├─ guard          empty / wildcard-only queries short-circuit
  ├─ route          (optional) LLM classifies intent, adjusts params
  ├─ entity detect  short name-like queries (<60 chars) match an entity first
  ├─ pod scope      resolve which pods this search may see
  │
  ├─► embed query ──► pgvector cosine  ─┐
  └─► to_tsquery  ──► tsvector BM25    ─┴─► RRF (K=20)
  │
  ├─ activation     ACT-R recency × frequency re-rank
  ├─ hebbian        co-retrieval boosts
  ├─ graph          (optional) expand one edge out, re-rank merged set
  ├─ floor          drop below minFactSimilarity
  ├─ visibility     filter by agent / device scope
  └─► top-K

Query routing

With routing on, an LLM first classifies the query’s intent and adjusts the search before it runs: whether to expand through the graph, how wide the result limit should be, which categories to filter to, and whether there’s a point-in-time constraint (“what did we decide before the migration”).

It costs one extra LLM call. The hook path uses it; interactive sigil search doesn’t unless you pass --route.

Entity-first search

When a query is short and looks like a name — under 60 characters, name-shaped — Sigil detects the entity first and searches outward from it, rather than treating “Priya” as a bag of characters to embed.

For “what do I know about X” questions this is dramatically better than similarity alone, because the answer isn’t textually similar to the question at all.

Graph enhancement

Optional. Takes the entities mentioned in the top results, pulls in facts connected to them through the entity graph, and re-ranks the merged set.

It finds things that share no vocabulary with your query but sit one edge away — you ask about the payment flow and get back a fact about a service your query never named.

Worth it for exploratory questions. Noise for precise ones, which is why it’s off by default and the router turns it on when it thinks the query wants it.

Floors

Two thresholds, deliberately different.

Setting	Default	Applies to
memory.minFactSimilarity	0.45	Any search — below this, results are dropped entirely
memory.injectionFloor	0.6	Automatic hook injection only

The injection floor is higher on purpose. Putting weak matches into every prompt is worse than putting nothing there — it costs tokens and it teaches the model to distrust the context block.

Namespaces and scope

sigil search "deploy" --namespace=engineering
sigil search "deploy" --all-namespaces
sigil search "retry" --scope                    # active project + session pods only

Scope is a ranking and visibility boundary, not a security one.

Synthesis

--synthesize (on by default via search.synthesize) hands the retrieved facts and raw chunks to an LLM, which writes an answer instead of returning a list. Chunks are forced into the result set when synthesis is on, so the synthesizer has material rather than only extracted atoms.

Turn it off when you want to see what’s stored rather than a paraphrase of it.

Performance

Search latency (avg / p95)	33ms / 61ms, local Postgres
Embedding latency	26ms
Tokens injected per prompt	~1.5K

Hook hot-path latency is higher on a cold start — 200–400ms for the first invocation, covering Node start, connection, and search — then warm. The daemon exists to keep it warm.

If that first-hit cost is unacceptable, remove the UserPromptSubmit hook from ~/.claude/settings.json and rely on the zero-latency hot-context snapshot plus on-demand MCP search.

Benchmarks

LongMemEval oracle split, n=100, OpenAI top-quality stack:

Metric	Result
R@1 / R@3 / R@10	100% / 100% / 100%
Answer correctness (LLM-judged)	43%
Cost / wall time	$0.21 / 37 min

Read the caveats

Oracle is the easy split. n=100 is small. The per-question haystack is ~25 chunks, so retrieval is nearly saturated and the number says less than it looks. Answer correctness is bottlenecked by the judge model’s temporal reasoning, not by retrieval. These aren’t comparable to Mem0 / Zep / Letta runs without identical methodology. Full write-up and reproducible scripts are in eval/longmemeval.

Sigil’s architectural work — Hebbian edges, ACT-R activation, pod-aware blending — shows up at 10K+ chunks, not at 25.

Reading a ranking

sigil why "how do we handle webhook retries"

Per fact: fused RRF score, Hebbian boost, category, importance, confidence, and the pods it came through. See Debugging retrieval.