The two-phase extraction pipeline

ZettelForge's two-phase pipeline turns source text into a small set of durable facts, then reconciles each fact with nearby memory. You use it through MemoryManager.remember(evolve=True), remember_with_extraction(), or the OSS zettelforge_remember MCP tool, which enables evolve by default.

The pipeline reduces duplicate, stale, and low-value notes. It does not provide lossless report ingestion in ZettelForge 2.8.0.

Extraction remains in progress

FactExtractor sends only the first 2,000 characters of each input or report chunk to the LLM. An empty or unparseable extraction response produces no facts. remember(evolve=True) then stores the original input through the direct path, but remember_report() stores nothing for the affected chunk. Use a direct, non-evolved write when every source character must be preserved.

Prerequisites: complete the Quickstart and configure a working provider with the LLM settings.

---

Why append-only memory degrades

An append-only memory system stores every report paragraph, repeated claim, and conversational aside. Semantic search still finds related notes, but three problems accumulate:

• Redundancy: repeated reporting competes for the same result slots.
• Contradiction: current and superseded assessments appear together.
• Noise: durable intelligence and temporary conversation receive equal weight.

The two-phase pipeline makes a decision before each extracted fact enters the main store. It keeps provenance through supersession instead of rewriting history in place.

Phase 1: extract candidate facts

FactExtractor treats the input as untrusted evidence and rejects recognized prompt-injection patterns before it calls the configured LLM provider. The extraction prompt asks for a JSON array of concise facts with importance values from 1 through 10.

[
  {"fact": "APT28 shifted to edge-device exploitation", "importance": 8},
  {"fact": "APT28 stopped using DROPBEAR", "importance": 7}
]

The parser clamps each importance value to the 1–10 range, sorts the facts from highest to lowest importance, and keeps the first max_facts. remember_with_extraction() then removes facts below min_importance.

Boundary	ZettelForge 2.8.0 behavior
Input sent to the extraction prompt	First 2,000 characters
max_facts for remember_with_extraction()	5
min_importance	3
max_facts for remember_report()	10 per chunk
Default report chunk target	3,000 characters, split at sentence boundaries
llm.max_tokens_extraction	2,500 output tokens

The 3,000-character report chunk target is larger than the 2,000-character extraction-prompt limit. The remaining portion of a chunk does not reach phase 1. This mismatch is the main reason report extraction remains marked in progress.

Phase 2: reconcile with memory

For every fact that passes phase 1, MemoryUpdater retrieves the three nearest notes in the same domain. It removes retrieved notes that contain recognized prompt-injection patterns, then asks the LLM to choose one internal operation.

Internal operation	Returned status	Storage effect
ADD	added	Store a new note with source_type="extraction"
UPDATE	updated	Store the refined fact and supersede the nearest safe note
DELETE	corrected	Store a correction note and supersede the nearest safe note
NOOP	noop	Store nothing

Here, DELETE means that the previous assessment loses authority. ZettelForge does not erase the old note.

If no safe similar note remains, phase 2 chooses ADD without an LLM decision. If the decision response is non-empty but invalid, ZettelForge retries once with an explicit JSON-format instruction. An empty response, a second parse failure, an invalid operation, or an exception defaults to ADD. That failure policy favors retaining a new fact over silently discarding it, but it can create duplicates.

How supersession preserves history

The UPDATE and DELETE paths write a new note first. mark_note_superseded() then records the new note in the old note's superseded_by field, adds the old ID to the new note's supersedes list, rewrites both notes, removes the old note from entity-index mappings, and writes a SUPERSEDES temporal edge from the new note to the old note.

Direct MemoryManager.recall() hides superseded notes by default. The OSS MCP recall handler explicitly sets exclude_superseded=False, so MCP recall can return both the current and historical assessments.

This split supports two legitimate views of the same corpus:

1. Current-only recall gives an operator the active assessment.
2. Historical recall preserves the chain of evidence and correction for review.

What callers receive

remember_with_extraction() returns one (MemoryNote | None, status) tuple for every processed fact. A NOOP result contains None; the other operations contain the new note.

remember(evolve=True) returns only the first tuple even when the pipeline processes several facts. If phase 1 returns no facts, it falls through to the direct write path and returns the directly stored note with status created.

remember_report() returns the combined tuples from all chunks. It returns an empty list when every chunk produces no qualifying facts, and it does not perform the direct-write fallback.

The MCP zettelforge_remember handler returns the first note ID, status, and up to ten extracted entities. When the first operation is NOOP, the note ID is null and the entity list is empty.

Failure behavior determines the safe write path

Extraction failures have different consequences:

Failure	Observed fallback
The extraction LLM call raises an exception	Return the first 500 input characters as one importance-5 candidate; the importance filter then determines whether it reaches phase 2
The extraction response is empty or cannot be parsed	Return no facts
remember(evolve=True) receives no facts	Store the original input through the direct path
remember_report() receives no facts for a chunk	Store nothing for that chunk
The phase-2 decision fails	Choose ADD

Use evolved writes when deduplication and correction matter and your inputs fit inside the verified extraction boundary. Use remember(evolve=False) or another direct-ingestion path when complete source preservation matters more than LLM reconciliation.

The design deliberately preserves history: new evidence can replace the current assessment without destroying the record that explains how the assessment changed.