Deduplication (AUDM)

The failure mode every memory system has is accumulation. Save “we use Postgres” fifty times across six months in five different phrasings, and every one of them competes for the same retrieval slot — splitting the ranking and crowding out everything else.

AUDM is the pass that prevents it. Every write goes through it.

Four decisions

Decision	When	Result
Add	Nothing sufficiently similar exists	A new fact row
Update	A near-match exists and this refines it	The existing fact is revised in place
Delete	This contradicts an existing fact	The old fact is superseded, not removed
Merge	Two facts are really one statement	Consolidated into one

How the decision is made

Similarity between the incoming fact and existing ones is computed, then compared against three thresholds.

similarity ≥ 0.88   →  skip           "you already know this"
0.78 – 0.88         →  ask the model  AUDM decides which of the four
0.72 – 0.78         →  supersede eligible
< 0.72              →  add

Setting	Default	Meaning
memory.skipThreshold	0.88	Above this, treat as a duplicate and skip entirely
memory.ambiguousThreshold	0.78	Above this, hand the decision to the LLM
memory.supersedeThreshold	0.72	Floor for considering a supersession
memory.supersedeScanLimit	8	How many existing candidates to compare against

The band between ambiguousThreshold and skipThreshold is where the model earns its keep. Two facts at 0.82 similarity might be the same statement reworded, or might be a genuine refinement — that’s not a question a number can answer.

Everything above skipThreshold is decided without an LLM call, which is what keeps write cost bounded.

Supersession, not deletion

When a new fact contradicts an old one, the old one is marked superseded and linked to its replacement. It stops being retrieved. It is not removed.

That’s a deliberate choice with three payoffs:

• Provenance survives. You can ask what you believed before the migration.
• Point-in-time queries work. Query routing can detect a temporal constraint and restrict retrieval to what was valid then.
• Mistakes are recoverable. A bad supersession is visible and reversible in a way a delete is not.

"Retries go through Redis pubsub"        [superseded 2026-03-11]
        ↓ superseded_by
"Retries run through the Postgres job table with SKIP LOCKED"   [current]

Where it runs

Path	AUDM
Stop hook (Claude Code)	Yes
sigil remember	Yes
MCP remember	Yes
sigil ingest — extracted facts	Yes
PostToolUse observations	SHA-256 exact dedup over a 5-minute window, then AUDM

PostToolUse gets the extra hash pass because a loop of edits produces near-identical observations at high volume, and a cheap exact-match filter is the right tool before spending an LLM call.

What you see

sigil remember "We use Postgres LISTEN/NOTIFY for events"

Remembered. (1 new)

Say it again in different words:

sigil remember "Event delivery goes through Postgres LISTEN/NOTIFY, not Redis"

Already known.

That’s not a failure. It’s the system telling you it recognized the statement — and it’s the reason a thousand saves don’t become a thousand rows.

Batched saves report a combined result: Remembered. (2 new, 1 updated, 1 already known).

Tuning

Duplicates are piling up. Lower memory.skipThreshold so more candidates are treated as the same fact. This is the common adjustment on a local LLM, which is weaker at recognizing that two differently-worded facts mean the same thing.

Distinct facts are being wrongly merged. Raise memory.skipThreshold. Rarer, and usually a sign your facts are too vaguely worded to be distinguishable in the first place — the fix is better facts, not a higher threshold.

Contradictions aren’t being caught. Lower memory.supersedeThreshold, or raise memory.supersedeScanLimit so more existing facts are compared against.

All four live in Settings → Advanced. See Configuration.

These fail quietly

A wrong threshold degrades memory rather than throwing an error. Too high and near-duplicates accumulate over months. Too low and real distinctions get collapsed. Change one at a time, and check with sigil facts —limit=100 a week later.

The decision model

AUDM decisions are the highest-value LLM call in the system — getting them wrong compounds over every future write. If you’re routing tasks to different models, this is the one worth spending on:

llm.extractionModel = openrouter:qwen/qwen3.5-flash              # cheap, per chunk
llm.decisionModel   = openrouter:anthropic/claude-sonnet-latest  # smart, per decision

Extraction is called per chunk and is volume work. AUDM is called per candidate and is judgement work. The split is worth configuring.

Checking it’s working

sigil facts --limit=100

Scan for variations on one idea. A handful is normal — genuinely distinct facts about the same topic. A dozen phrasings of the same sentence means skipThreshold is too high, or extraction is producing facts too vague to be recognized as duplicates.

The dashboard’s Activity screen shows each AUDM decision with the model that made it and what it chose, which is faster than inferring it from the fact list.