Skip to content

RFC-001: Conversational Entity Extractor

Field Value
Author Patrick Roland
Status Partially Implemented
Created 2026-04-09
Last updated 2026-07-27
ZettelForge version v2.7.0
Related RFCs RFC-002 Universal LLM Provider

Context

ZettelForge's entity extractor recognizes CTI entities: CVEs, APT groups, tools, campaigns, IOCs. The LOCOMO benchmark tests conversational memory — person names, locations, hobbies, life events, and temporal references. Those entity types were invisible to the regex-only extractor.

At v1.5.0, ZettelForge scored 15% on LOCOMO. Root cause analysis identified entity extraction mismatch as the primary blocker: retrieved context contained the answer (75.9% keyword presence), but graph traversal returned nothing because no recognized entities matched LOCOMO queries. Multi-hop and temporal accuracy were 0%.

Proposal

Replace regex-only entity extraction with a hybrid pipeline:

  • Regex fast-path for CTI and IOC types (reliable, deterministic, no LLM required)
  • LLM NER for conversational types (person, location, organization, event, activity, temporal) where regex fails

This gives 19 entity types total: 6 CTI, 7 IOC, 6 conversational.

Target: 80%+ LOCOMO accuracy.

What shipped

Entity types (Steps 1–3 — complete)

EntityExtractor in entity_indexer.py is the single source of truth for all extraction. NoteConstructor.extract_entities() delegates to it. The 19-type schema is live:

Category Entity types Extraction method
CTI cve, intrusion_set, actor, tool, campaign, attack_pattern Regex
IOC (STIX cyber observables) ipv4, domain, url, md5, sha1, sha256, email Regex
Conversational person, location, organization, event, activity, temporal Regex fallback plus background LLM NER when enabled

The IOC types were not in the original proposal — they were added during implementation to support CTIBench and real-world CTI workflows.

For entity type reference and the get_entity_relationships() / traverse_graph() API, see KG Edge Schema and Memory Manager API.

Hash false-positive filter (complete)

Hash IOC patterns (md5/sha1/sha256) match any 32/40/64-character hex string, producing false positives on git commit SHAs and code assignments. EntityExtractor._CODE_CONTEXT_PATTERN detects code or VCS context per line and excludes all hex strings on matching lines from final results. This is a compiled re.VERBOSE pattern that handles assignments, commit entries, merge lines, code fences, and function definitions.

_PERSON_PATTERN and _LOCATION_PATTERN (complete, limited)

Two regex patterns handle conversational entities without an LLM call:

  • _PERSON_PATTERN matches dialogue format Name: text (e.g., transcript speaker labels), with a stopword filter excluding 40+ common words and day/month names.
  • _LOCATION_PATTERN matches 40+ major world cities by name.

These are supplementary. They do not cover implicit references ("my friend from college") or locations not in the hardcoded list.

extract_llm() and LLM NER enrichment (complete; background path active)

EntityExtractor.extract_llm() is implemented and functional when called directly. It calls the configured LLM provider with a structured JSON prompt, isolates input to 2,000 characters, uses get_config().llm.max_tokens_ner (default 2500) with temperature=0.0, and retries once on JSON parse failure with json_mode=True and temperature=0.3. On exception, it returns an empty dict and logs exc_info=True.

The foreground remember() path still indexes the note with extract_all(use_llm=False) for deterministic writes. After that write, MemoryManager.remember() dispatches an llm_ner enrichment job when both enrichment.enabled and llm_ner.enabled are true. Both default to true and can be disabled with ZETTELFORGE_ENRICHMENT_ENABLED=false or ZETTELFORGE_LLM_NER_ENABLED=false. _run_llm_ner() calls extract_llm(), then writes any new entity mappings into the storage backend and the legacy JSON EntityIndexer.

Benchmark ingestion disables enrichment with ZETTELFORGE_ENRICHMENT_ENABLED=false, so LOCOMO runs do not exercise LLM NER unless you intentionally override that benchmark default.

EntityIndexer persistence (complete)

EntityIndexer maintains an inverted index: {entity_type -> {entity_value -> Set[note_id]}}. Writes are batched via a 5-second threading.Timer (deferred flush). An atexit handler flushes on process exit. build() reindexes all notes synchronously with regex-only extraction.

As of v2.2.0, entity data exists in two stores: entity_index.json (the primary runtime store) and a SQLite entity_index table. The two are not kept in sync by a single authoritative writer. This is a known inconsistency.

Tests

tests/test_conversational_entities.py has 27 tests across 8 classes (5 more than the 22 named at RFC write time; TestFreeTextPersonNotExtracted was added after the adversarial review):

Class Tests Notes
TestRegexExtraction 5 CVE, actor, tool, campaign, no-match
TestLLMExtraction 3 Runs against the mock LLM provider
TestHybridExtraction 2 Runs against the mock LLM provider
TestNERParsing 3 JSON parse logic with mocked output
TestNoteConstructorDelegation 2 Delegation wiring
TestInferEntityType 5 CVE, APT, tool, type hints, unknown
TestEntityIndexerConversational 3 All types present, add/lookup, cross-type search
TestFreeTextPersonNotExtracted 4 Stopword filter regression tests (added post-RFC)

Additional coverage now exists outside this file: tests/test_basic.py::TestEntityExtractor covers IOC extraction and the git-SHA hash false-positive filter, and tests/test_entity_indexer_races.py covers the RFC-001 Warning 4, 5, and 6 fixes for atomic save, remove_note(), and concurrent save/mutation. The remaining test gap is live-provider LLM NER quality; current LLM NER tests prove parser and wiring behavior with a mock provider, not extraction quality from a real model.

What did not ship

Step 4 — production SynthesisGenerator in benchmark. locomo_benchmark.py now calls a benchmark-local _synthesize_answer() helper after retrieval. That helper uses Ollama's /api/generate endpoint with SYNTH_MODEL defaulting to qwen2.5:3b, then falls back to relevant-sentence extraction when the LLM call fails. The original RFC goal of wiring the production SynthesisGenerator into the benchmark has not shipped.

Foreground LLM NER activation. The deterministic foreground indexing path still uses extract_all(use_llm=False). LLM NER activation now exists as a background enrichment job, not as an inline use_llm=True call inside remember() or NoteConstructor.

Batch LLM NER. LLM NER jobs run one note at a time through the enrichment worker. No batch API or batched prompt path has shipped.

EntityIndexer backend unification. The JSON/SQLite dual-write inconsistency has not been resolved.

LOCOMO benchmark results

The RFC targeted 80%+ LOCOMO accuracy. Actual results:

Version Score Judge Primary driver
v1.5.0 15% Keyword overlap Baseline (regex-only CTI)
v2.0.0 18% Keyword overlap Entity schema expansion
v2.1.1 22% Ollama cloud model Supersession perf fix, file locking, cloud judge
v2.7.0 7% Keyword judge Regression vs v2.1.1 Ollama judge; methodology change

The 80% target was not achieved. Analysis:

  • Entity extraction alone is insufficient. The benchmark now has a local focused-answer helper, but it does not use the production SynthesisGenerator, and the fallback still evaluates extracted context snippets when no synthesis LLM is available.
  • The 3B model ceiling. Qwen2.5-3B handles common entities adequately but struggles with implicit references and abstract events. LOCOMO relies heavily on these.
  • LLM NER is activated only through background enrichment and is disabled during the deterministic benchmark path. The core RFC value proposition, LLM-powered conversational NER improving multi-hop and temporal accuracy, remains unmeasured in the LOCOMO benchmark.
  • Most of the v2.1.1 gain came from performance fixes and a better judge, not from entity extraction.

The v2.7.0 7% keyword-judge baseline is not directly comparable to the v2.1.1 22% Ollama-judge result. The methodology change (keyword judge vs LLM judge) accounts for most of the apparent regression. See Reproduce Benchmarks for how to replicate either result.

Decision

Accepted (2026-04-09). Adversarial review completed 2026-04-16. Two blockers identified and actioned: load() log event name corrected; attack_pattern regex capture group fixed. Remaining findings documented as open issues.

Status as of v2.7.0: Partially Implemented. Steps 1–3 are complete. LLM NER activation exists as background enrichment. Step 4's production SynthesisGenerator benchmark integration is not complete.

Adversarial review findings (2026-04-16)

# Severity Finding Status
1 BLOCKER use_llm=True never called — LLM NER is dead code Fixed as background enrichment; foreground path remains regex-only
2 BLOCKER load() logs entity_index_save_failed instead of load_failed Fixed
3 WARNING attack_pattern regex lacked capture group Fixed
4 WARNING save() truncates before flock — race condition Fixed
5 WARNING remove_note() deletes entity type keys Fixed
6 WARNING _flush_sync() not thread-safe on self.index Fixed
7 WARNING extract_all(use_llm=True) overwrites regex results Fixed
8 WARNING No tests for IOCs, hash filter, remove_note, search_entities Fixed except live-provider LLM NER quality coverage
9 WARNING Root cause analysis omitted LLM NER not activated Fixed in RFC
10 NIT Test count stated as 18, actual was 23 (now 27 in v2.7.0) Fixed in RFC

Recommendations for future work

  1. Measure background LLM NER impact on LOCOMO with enrichment enabled and a reachable real LLM provider.
  2. Replace the benchmark-local _synthesize_answer() helper with the production SynthesisGenerator, then re-run LOCOMO with the same judge and dataset.
  3. Add mock-based NER regression tests that cover extract_llm() without a live model. TestNERParsing already covers parse logic; extend to the full extraction path.
  4. Unify EntityIndexer storage to write through StorageBackend and eliminate the JSON/SQLite dual-write inconsistency.
  5. Evaluate RFC-002 (Universal LLM Provider) impact: once a larger model is accessible through a stable provider interface, re-run LOCOMO to measure the NER quality ceiling independent of the 3B model constraint.