Architecture deep dive¶

Use this page when you need to trace a note through ZettelForge or decide where an integration belongs. ZettelForge coordinates governance, note construction, hybrid storage, retrieval, and optional LLM synthesis through MemoryManager.

This reference describes ZettelForge v2.8.0, the self-hosted open-source product. The verified release metadata declares an MIT license and Python 3.10 or later. ThreatRecall.ai is the SaaS product; it is outside the runtime boundary described here.

Prerequisites¶

Read Configuration Reference before you change providers or storage paths. Use Memory Manager API for method signatures and Retrieval Policies for intent weights.

Runtime boundaries¶

MemoryManager owns the main process boundary. It initializes the durable note store, the vector index, entity extraction, retrieval, governance, consolidation, memory defense, and a background enrichment worker.

Boundary	v2.8.0 implementation	Responsibility
Coordinator	`MemoryManager`	Runs write, recall, synthesis, telemetry, and lifecycle paths
Durable records	SQLite at `~/.amem/zettelforge.db` by default	Stores notes, graph nodes, graph edges, entity mappings, and enrichment jobs
Vector index	LanceDB at `~/.amem/vectordb` by default	Stores one vector table per domain and serves semantic candidates
Entity layer	`EntityIndexer` and `AliasResolver`	Extract and normalize entities, then map them to notes
Retrieval	`VectorRetriever`, `GraphRetriever`, and `BlendedRetriever`	Rank semantic and graph candidates, then add entity matches
Governance	`GovernanceValidator` and `MemoryAnomalyGate`	Enforce input limits and prompt-injection checks, apply optional PII handling, and evaluate memory anomalies
Generation	`SynthesisGenerator`	Retrieves context and asks the configured LLM provider for a structured answer
Integration surface	MCP server over JSON-RPC 2.0 on standard input and output	Exposes the memory operations to compatible clients

The storage backend factory always constructs SQLiteBackend in v2.8.0. If you set the legacy JSONL backend selector, the factory logs a warning and still uses SQLite.

Write path¶

When you call remember(), ZettelForge runs these stages in order:

GovernanceValidator validates the input. It enforces the configured content limit, rejects detected prompt-injection patterns, and applies PII handling only when you enable the optional PII dependency and configuration.
If you set evolve=True, the fact extraction and update path can add, update, correct, or skip facts. If it produces no fact above the configured threshold, the normal write path continues.
NoteConstructor builds a MemoryNote, derives lightweight context and keywords, extracts entities, and generates an embedding.
MemoryAnomalyGate evaluates the note under the configured audit, block, or quarantine mode.
SQLiteBackend writes the note to the notes table.
MemoryStore writes the vector into the domain-specific LanceDB table when LanceDB is available. A vector-index failure is logged and does not roll back the SQLite write.
The entity path resolves aliases, updates the in-process index, and persists entity-to-note mappings in SQLite.
Consolidation and supersession checks run before heuristic graph nodes and edges are written to SQLite.
When enrichment is enabled, the coordinator dispatches causal extraction, LLM entity extraction, and neighbor evolution jobs. sync=False queues the work on the background worker; sync=True runs it inline.
ZettelForge emits structured operation telemetry and returns the note with its write status.

This ordering creates an important failure boundary: SQLite holds the durable note record, while LanceDB is a derived search index. Monitor vector-index warnings because a successful note write does not prove that semantic indexing succeeded.

Read path¶

When you call recall(), ZettelForge runs a bounded hybrid retrieval pipeline:

The prompt-injection guard validates the query.
IntentClassifier selects a retrieval policy.
EntityIndexer extracts query entities, AliasResolver normalizes them, and a fan-out filter removes low-signal entities.
VectorRetriever queries LanceDB. If that path fails or returns no candidates, it falls back to in-memory cosine search.
GraphRetriever performs breadth-first traversal against the same SQLite graph that the write path populated. It does not use the separate process-wide JSONL graph helper for recall().
BlendedRetriever min-max normalizes vector and graph scores, applies the intent policy weights, and ranks the merged candidates.
Causal queries can add candidates from forward and incoming causal edges. Entity lookup can add exact entity matches.
When reranking is enabled and more than one result remains, the cross-encoder reranks only the configured candidate head. A reranker failure preserves the prior order.
The coordinator removes superseded and expired notes unless you request them, caps the result at k, updates access metadata, and emits telemetry.

Vector and graph retrieval run sequentially in the v2.8.0 implementation. Treat diagrams or older descriptions that call these stages parallel as stale.

Storage model¶

SQLite is the source of record¶

SQLiteBackend.initialize() enables write-ahead logging, normal synchronous mode, foreign keys, and a five-second busy timeout. It creates five tables:

Table	Stored data
`notes`	Content, provenance, semantic fields, embedding metadata, confidence, persistence, evolution, and links
`kg_nodes`	Normalized graph nodes keyed by entity type and value
`kg_edges`	Directed relationships, provenance, edge type, and properties
`entity_index`	Entity-to-note mappings
`enrichment_jobs`	Background job state and attempt metadata

SQLite uses a process-local reentrant lock for writes. Closing the backend checkpoints and truncates the write-ahead log before the connection closes.

LanceDB is a rebuildable index¶

MemoryStore creates ~/.amem/vectordb by default and lazily opens LanceDB. Each domain uses a table named notes_<domain>. Rows contain the note ID, vector, a bounded content excerpt, semantic context, keywords, tags, and creation time.

VectorRetriever resolves the full note through SQLite after LanceDB returns an ID. This split keeps the vector row small while preserving the complete record in the source-of-record database.

The JSONL graph is a separate API¶

The public KnowledgeGraph helper persists kg_nodes.jsonl and kg_edges.jsonl. It supports direct graph and temporal queries, but MemoryManager.recall() wraps the active storage backend with StoreGraphSource and reads SQLite graph tables instead. Do not assume that writing through the standalone JSONL helper updates recall results.

Failure and concurrency behavior¶

Condition	Observed v2.8.0 behavior
SQLite initialization fails	`MemoryManager` cannot establish its durable store
LanceDB connection or indexing fails	ZettelForge logs the error; a completed SQLite write remains durable
Vector retrieval fails	`VectorRetriever` falls back to in-memory cosine search
Reranking fails	Recall keeps the blended order
Recall exceeds its configured timeout	Recall logs the timeout and returns an empty list
Background queue is full	The dispatch path logs the condition; the completed fast-path write remains durable
Process exits normally	Registered exit handlers drain enrichment work before closing SQLite

The enrichment queue has one daemon worker per MemoryManager instance. The MCP server creates its process-wide manager lazily on the first tool call, so importing the server or listing its tool definitions does not open storage or start the worker.

Security boundaries¶

The write path applies deterministic prompt-injection checks before storage. The recall and synthesis paths apply the same class of guard before untrusted text reaches retrieval or an LLM prompt. Optional PII handling remains disabled until you install its dependency and enable it.

Memory anomaly defense is enabled in audit mode by the v2.8.0 dataclass defaults. Audit mode records a decision without blocking the write. Change the mode deliberately if you need block or quarantine behavior, and verify the effective configuration in your environment.

ZettelForge does not encrypt SQLite, LanceDB, or JSONL files itself. Use operating-system or volume encryption when you need encryption at rest.

Integration boundary¶

The MCP server speaks JSON-RPC 2.0 over standard input and output and reports MCP protocol version 2024-11-05. Its handlers delegate memory operations to the process-wide MemoryManager.

Native OpenCTI pull and push synchronization is in progress. The registered sync tool does not provide a working sync path in the verified ZettelForge v2.8.0 base release. Use MCP Protocol Reference for the currently verified tool contract.

Design trade-offs¶

SQLite and LanceDB separate durable records from semantic indexing. You can recover from an index failure, but you must detect and rebuild drift.
Intent-weighted fusion combines semantic similarity with explicit relationships. It costs more than a single lookup and can degrade to the surviving signal when one retriever fails.
The background worker shortens the foreground write path. Derived causal and LLM-enriched data becomes eventually consistent with the durable note.
The standalone JSONL graph preserves a direct graph API, while the main recall path stays scoped to the manager's SQLite store. Mixing those paths creates divergent graph views.
Local files keep self-hosting simple. They also place backup, access control, encryption, and multi-process coordination responsibilities on you.

Source trail¶

This page was verified against the ZettelForge v2.8.0 release and its tagged source tree:

memory_manager.py for initialization, write, recall, synthesis, and worker lifecycle
sqlite_backend.py for the durable schema and connection settings
memory_store.py and vector_retriever.py for vector indexing and fallback search
graph_retriever.py and knowledge_graph.py for the two graph paths
governance_validator.py and prompt_injection_guard.py for input controls
mcp/server.py for JSON-RPC and lazy manager initialization