Module inventory

Complete reference of all Python modules in ZettelForge v2.7.0. Use this page to understand what each module does before reading or contributing code.

At a glance

Metric	Value
Python files in src/zettelforge/	107
Top-level packages	17
Public API exports (__all__)	29
Python files in web/	3
JavaScript files in web/ui/	17
Most-imported module	log.py (47 dependents)

Prerequisites: ZettelForge 2.7.0 source tree. All paths below are relative to the repo root.

---

Core memory layer

src/zettelforge/memory_manager.py — 1,972 lines

Primary interface for all memory operations. Every recall, ingest, synthesize, and graph query goes through MemoryManager. Orchestrates the two-phase pipeline (fact extraction → note construction), retrieval (vector + graph + blend), and synthesis.

Key class: MemoryManager

Key methods: remember(), recall(), recall_actor(), synthesize(), ingest_relationship(), get_entity_relationships(), traverse_graph(), stats().

src/zettelforge/__init__.py

Public API surface. Exposes 29 names in __all__:

ENTITY_TYPES, RELATION_TYPES,
BlendedRetriever, Edition, EditionError, ExtractedFact, FactExtractor,
GraphRetriever, IntentClassifier, KnowledgeGraph, MemoryManager, MemoryNote,
MemoryUpdater, NoteConstructor, QueryIntent, ScoredResult,
SynthesisGenerator, SynthesisValidator, UpdateOperation, VectorRetriever,
edition_name, get_edition, get_intent_classifier, get_knowledge_graph,
get_memory_manager, get_synthesis_generator, get_synthesis_validator,
is_community, is_enterprise

src/zettelforge/__main__.py

CLI entry point. Invoked by python -m zettelforge.

src/zettelforge/demo.py

Interactive demo showing CTI memory in action. Run with python -m zettelforge.demo.

---

Storage layer

src/zettelforge/storage_backend.py

Abstract base class (StorageBackend) that all storage backends implement. Defines the full contract for note operations, KG operations, entity mapping, and full-text search.

Key classes: StorageBackend (ABC), BackendClosedError

src/zettelforge/sqlite_backend.py — 969 lines

SQLite implementation with WAL mode. Default backend for ZettelForge OSS.

Key class: SQLiteBackend

Tables: notes (primary storage), kg_nodes, kg_edges. Features: WAL mode for concurrent reads, full-text search indexes, ACID transactions.

src/zettelforge/memory_store.py — 445 lines

JSONL + LanceDB hybrid storage. Alternative to SQLite; used for direct graph-class access paths.

Key class: MemoryStore

src/zettelforge/backend_factory.py

Creates the appropriate StorageBackend based on config. Call get_storage_backend(config) to get a live backend instance.

src/zettelforge/vector_memory.py

Cross-session semantic memory using LanceDB and Nomic embeddings. Handles chunking (512 tokens, 128 overlap), embedding generation, and content-hash deduplication.

Key class: VectorMemory

src/zettelforge/lance_maintenance.py

Periodic LanceDB version-cleanup daemon (RFC-009 Phase 1.5). Compacts old LanceDB versions to reclaim disk space.

Key class: LanceVersionMaintenance

---

Retrieval layer

src/zettelforge/vector_retriever.py

Vector similarity search over the LanceDB index. Falls back to in-memory cosine similarity when the index is unavailable.

Key class: VectorRetriever

Defaults: similarity threshold 0.15, IVF_FLAT index, entity boost for exact entity matches.

src/zettelforge/graph_retriever.py

Knowledge graph traversal for note retrieval. BFS up to max_depth=2 hops; score decays with distance: score = 1.0 / (1.0 + hop_distance).

Key classes: GraphRetriever, GraphSource, StoreGraphSource, ScoredResult

src/zettelforge/blended_retriever.py

Fuses vector and graph retrieval results. Two fusion methods:

• blend() — min-max normalized score fusion (default since v2.3.1)
• blend_rrf() — Reciprocal Rank Fusion with rrf_k=60

Key class: BlendedRetriever

src/zettelforge/intent_classifier.py

Adaptive query routing (MAGMA-style). Classifies each query into one of five intents using keyword matching first, then LLM fallback.

Key class: IntentClassifier
Key class: QueryIntent — enum values: FACTUAL, TEMPORAL, RELATIONAL, CAUSAL, EXPLORATORY

---

Knowledge graph layer

src/zettelforge/knowledge_graph.py — 534 lines

JSONL-based knowledge graph for direct graph access (outside the SQLite backend path). Stores nodes and edges in kg_nodes.jsonl and kg_edges.jsonl. Supports BFS traversal, temporal indexing, and in-memory caching.

Key class: KnowledgeGraph

src/zettelforge/ontology.py — 659 lines

Typed entity system with ontology constraints. Validates entity types and relationships against the ZettelForge ontology.

Key classes: OntologyValidator, TypedEntityStore

src/zettelforge/entity_indexer.py — 651 lines

Entity extraction and indexing. Extracts CTI entities using 13 regex patterns and up to 6 LLM NER types.

Key classes: EntityExtractor, EntityIndexer

Entity categories: - CTI: cve, intrusion_set, actor, tool, campaign, attack_pattern - IOCs: ipv4, domain, url, md5, sha1, sha256, email - Conversational (LLM): person, location, organization, event, activity, temporal

src/zettelforge/alias_resolver.py

Resolves entity aliases so queries for APT28 also surface notes tagged Fancy Bear or STRONTIUM. Requires an LLM provider for the enrichment path; offline, only exact canonical matches succeed.

Key class: AliasResolver

---

Note construction and schema

src/zettelforge/note_schema.py

Pydantic schemas for MemoryNote and its nested models.

Key classes: MemoryNote, Content, Semantic, Embedding, Links, VulnerabilityMeta

src/zettelforge/note_constructor.py

LLM-powered note enrichment. Assembles MemoryNote objects from raw content, runs entity extraction, generates IDs, manages timestamps, and computes content hashes.

Key class: NoteConstructor

src/zettelforge/fact_extractor.py

Phase 1 of the two-phase pipeline. Extracts discrete facts from raw content using the LLM.

Key classes: FactExtractor, ExtractedFact

src/zettelforge/memory_updater.py

Phase 2 of the two-phase pipeline. Determines whether to add, update, delete, or no-op on existing notes based on new incoming content.

Key classes: MemoryUpdater, UpdateOperation (ADD | UPDATE | DELETE | NOOP)

src/zettelforge/memory_evolver.py

A-Mem-inspired neighbor refinement. Compares new intelligence against existing notes and applies evolution decisions; handles contradictions and supersession tracking.

Key class: MemoryEvolver

src/zettelforge/consolidation.py — 490 lines

GAM-style consolidation layer. Detects semantic shift in a note cluster and merges or archives stale notes.

Key classes: SemanticShiftDetector, ConsolidationEngine, ConsolidationMiddleware

---

Synthesis layer

src/zettelforge/synthesis_generator.py

RAG answer synthesis (Phase 7). Takes retrieved notes and generates a structured answer using the configured LLM.

Key class: SynthesisGenerator

Formats: direct_answer (OSS default), synthesized_brief, timeline_analysis, relationship_map (the latter three require ThreatRecall.ai SaaS).

Context window: up to 10 notes, 500 characters per note, 3,000 tokens total.

src/zettelforge/synthesis_validator.py

Validates synthesis output against schema, checks confidence thresholds, and verifies source attribution.

Key class: SynthesisValidator

---

LLM integration layer

src/zettelforge/llm_client.py

Unified LLM interface. Dispatches to the configured provider via generate() and generate_structured().

src/zettelforge/llm_providers/ — 7 files

File	Purpose
base.py	LLMProvider protocol — contract all backends implement
local_provider.py	In-process llama-cpp-python GGUF provider (local)
ollama_provider.py	Ollama HTTP provider
litellm_provider.py	LiteLLM routing provider — supports OpenAI, Anthropic, Azure, and others
mock_provider.py	Deterministic mock responses for tests
registry.py	Thread-safe provider registry keyed by name
__init__.py	Package exports

---

Governance and security layer

src/zettelforge/governance_validator.py

Write-time governance validation. Enforces content size limits, TLP markings, and retention policies before any note is written.

Key classes: GovernanceValidator, GovernanceViolationError

src/zettelforge/prompt_security.py

Prompt-injection and retrieval-poisoning controls. Detects 7 injection pattern categories using deterministic regex; always active regardless of config.

Key classes: PromptSecurityFinding, PromptInjectionError

Pattern categories: direct_instruction_override, role_takeover, system_prompt_exfiltration, secret_exfiltration, tool_instruction_smuggling, retrieval_poisoning, role_delimiter_smuggling.

src/zettelforge/pii_validator.py

PII detection and redaction (RFC-013). Uses Microsoft Presidio when the zettelforge[pii] extra is installed; disabled by default.

Key classes: PIIDetection, PIIValidator, PIIBlockedError

src/zettelforge/memory_defense.py

Write-time memory poisoning defenses. MemoryAnomalyGate scores candidate notes for distributional anomaly using MemSAD (Memory Semantic Anomaly Detection). Default mode: audit (log only).

Key classes: MemoryAnomalyGate, MemoryAnomalyDecision, MemoryAnomalyError

src/zettelforge/tlp_policy.py

TLP normalization, propagation, and sharing decisions. Determines the maximum shareable TLP for each actor context. Default share-max: TLP:GREEN; full-access actors: TLP:RED.

Key class: TlpSharingError

---

Observability and telemetry

src/zettelforge/log.py

Structured logging. get_logger(name) returns a structlog logger bound to the module. Most-imported module in the codebase (47 dependents).

src/zettelforge/ocsf.py

OCSF v1.3 event emitters (GOV-012 compliant). Emits structured audit events for all recall, ingest, auth, and file operations.

Emitter functions: log_api_activity(), log_authorization(), log_file_activity(), and 4 others.

src/zettelforge/telemetry.py

Operational telemetry for recall and synthesis quality monitoring. Collects per-query metrics written to a daily JSONL file.

Key class: TelemetryCollector

src/zettelforge/observability.py

Observability helpers. Provides Observability class for metrics collection and export.

Key class: Observability

---

Configuration and edition

src/zettelforge/config.py

Configuration loader. Merges environment variables, config.yaml (working directory), config.yaml (project root), config.default.yaml, and hardcoded defaults — in that priority order.

Key classes: StorageConfig, TypeDBConfig, EmbeddingConfig, LLMConfig, LLMNerConfig, plus 9 others covering extraction, retrieval, synthesis, governance, lance, cache, logging, web, and OpenCTI.

src/zettelforge/edition.py

Edition detection. Inspects loaded extensions to determine whether the runtime is OSS or a ThreatRecall.ai SaaS deployment. Does not gate features itself; the result is consumed by is_enterprise() and is_community() callers.

Key classes: Edition, EditionError
Key functions: is_enterprise(), is_community(), edition_name(), get_edition()

src/zettelforge/extensions.py

Extension loader. Attempts to import optional packages and registers them for use by edition detection and the backend factory. If the package is absent, loading fails silently.

---

Utility modules

src/zettelforge/cache.py

In-memory caching with TTL and entry-count limits.

Key class: SmartCache

src/zettelforge/retry.py

Retry logic with exponential backoff.

Key class: RetryConfig

src/zettelforge/json_parse.py

Safe JSON extraction from LLM output. Handles partial output, code fences, and trailing text.

Key function: extract_json()

src/zettelforge/backup.py

Backup, restore, and validation helpers for local ZettelForge data.

Key classes: BackupError, BackupResult, RestoreResult

---

Detection rules packages

src/zettelforge/sigma/ — 6 files + 1 schema file

File	Purpose
__init__.py	Package exports
parser.py	YAML load + JSON-schema validation
entities.py	Sigma rule → entity/relation mapping
ingest.py	High-level Python ingest API
cli.py	CLI: python -m zettelforge.sigma.ingest <path>
tags.py	Namespaced tag → typed entity resolver
schemas/__init__.py	Vendored SigmaHQ JSON schemas (see NOTICE.md)

src/zettelforge/yara/ — 7 files + 1 schema file

File	Purpose
__init__.py	Package exports
parser.py	plyara wrapper for YARA rule parsing
entities.py	YARA rule → entity/relation mapping
ingest.py	High-level Python ingest API
cli.py	CLI: python -m zettelforge.yara.ingest <path>
tags.py	Inline tag → typed entity resolver
cccs_metadata.py	CCCS YARA metadata validator (clean-room re-implementation)
schemas/__init__.py	Vendored CCCS YARA schemas (see NOTICE.md)

src/zettelforge/detection/ — 4 files

File	Purpose
__init__.py	Package exports
base.py	DetectionRule supertype dataclass shared by Sigma and YARA
explainer.py	LLM-generated rule explanation
consumers.py	DetectionMatchConsumer protocol for deferred match-event hooks

---

OSINT package

src/zettelforge/osint/ — 6 files

File	Purpose
__init__.py	Package exports (RFC-016 / RFC-0001)
ontology.py	OSINT entity types and relationship taxonomy
entity_resolver.py	Canonical key normalization and alias index
executor.py	Collector executor and KG ingestion path
investigation.py	Named case scoping for OSINT enrichment data
transform_registry.py	Collector registry and CollectorTuple type

src/zettelforge/osint/collectors/

Collector implementations grouped by target category. Live collectors are Phase 1; stubs return empty lists until their respective phase ships.

File	Phase	Status
infrastructure/dns_collector.py	1	Live — A, AAAA, NS, MX records via dnspython
infrastructure/cert_collector.py	1	Live
infrastructure/whois_collector.py	1	Live
infrastructure/bgp_collector.py	1.5	Live
infrastructure/port_scanner.py	1.5	Stub
people/holehe_collector.py	2	Stub — returns [] when holehe not installed
people/hunter_collector.py	2	Stub
people/namechk_collector.py	2	Stub
tech/builtwith_collector.py	3	Stub
tech/wappalyzer_collector.py	3	Stub
social/hashtag_tracker.py	4	Stub
social/twitter_collector.py	4	Stub
breach/breach_directory.py	4	Stub
breach/hibp_collector.py	4	Stub

---

Integration packages

src/zettelforge/integrations/ — 3 files

File	Purpose
__init__.py	Package exports
langchain_retriever.py	LangChain retriever wrapper
crewai.py	CrewAI tool wrapper

src/zettelforge/mcp/ — 3 files

File	Purpose
__init__.py	Package exports
server.py	MCP server implementation — JSON-RPC 2.0 over stdio
__main__.py	Entry point: python -m zettelforge.mcp

MCP tools exposed (7 total):

Tool	Notes
zettelforge_remember	Store content
zettelforge_recall	Retrieve notes
zettelforge_synthesize	Generate RAG answer
zettelforge_entity	Entity lookup
zettelforge_graph	Graph traversal
zettelforge_stats	System statistics
zettelforge_sync	OpenCTI sync — returns 501 in OSS; ThreatRecall.ai SaaS only

---

Operator scripts

src/zettelforge/scripts/ — 6 files

File	Purpose
__init__.py	Package exports
backup_restore.py	CLI for backup, restore, and archive validation
compact_lance.py	One-shot LanceDB shard compaction
human_eval_sampler.py	Human evaluation sampler (US-004)
telemetry_aggregator.py	Summarize daily telemetry JSONL into operational metrics
telemetry_dashboard.py	Streamlit telemetry dashboard (RFC-007 / US-005)

---

Web layer

The web server and UI live in web/ at the repo root, separate from the src/zettelforge/ package.

web/ — Python files (3)

File	Lines	Purpose
app.py	4,401	FastAPI server — all 25 REST endpoints + SPA serving
auth.py	—	API key validation and loopback allowlist
mcp_server.py	—	MCP server adapter for web deployment

web/ui/ — JavaScript files (17)

Vanilla JS single-page application. No build step, no npm.

Library layer:

File	Purpose
js/lib/state.js	Reactive pub/sub state store
js/lib/api.js	Fetch wrapper with API key headers
js/app.js	SPA router and bootstrap

Components (6):

File	Purpose
js/components/header.js	Top bar with ZettelForge logo and stats
js/components/sidebar.js	Navigation sidebar
js/components/result-card.js	Memory note display row
js/components/spinner.js	Loading indicator
js/components/tabs.js	Tab switcher
js/components/toast.js	Toast notification

Views (8):

File	Purpose
js/views/dashboard.js	System health tiles and telemetry charts
js/views/search.js	Recall, synthesize, and remember interface
js/views/knowledge-graph.js	2D SVG force-directed graph
js/views/logs.js	Filterable log table with auto-refresh
js/views/ingest.js	Manual and bulk ingestion
js/views/entities.js	Paginated entity browser
js/views/history.js	Session activity timeline
js/views/configuration.js	Feature toggles and YAML config editor

Styles: web/static/css/design_tokens.css

---

Core dependency graph

memory_manager
├── storage_backend → sqlite_backend (default)
├── vector_retriever → vector_memory, entity_indexer
├── graph_retriever → knowledge_graph
├── blended_retriever → vector_retriever, graph_retriever
├── synthesis_generator → vector_retriever, llm_client
├── note_constructor → entity_indexer, note_schema
├── fact_extractor → llm_client, json_parse
├── memory_updater → note_schema
├── intent_classifier → llm_client
├── governance_validator → pii_validator, prompt_security, memory_defense, tlp_policy
└── log (imported by 47 modules)