Reproduce benchmarks

Use this page to run the benchmark scripts that ship in the ZettelForge source checkout. Start with CTI retrieval. It is the domain benchmark, uses an embedded CTI corpus, and does not need an external dataset.

The CTI retrieval and CTIBench commands below are verified against ZettelForge 2.8.0. LOCOMO, RAGAS, MemPalace, and MemoryAgentBench are in progress for this how-to. The v2.8.0 prompt-injection guard blocks the three dataset-backed ZettelForge harnesses, and the MemPalace dependency is not installed here.

Prerequisites

• A ZettelForge 2.8.0 source checkout.
• Python 3.10 or newer with the ZettelForge runtime dependencies installed.
• A shell at the repository root.
• For LOCOMO and RAGAS, locomo10.json at the script default path ~/.openclaw/workspace-nexus/Locomo-Plus/data/locomo10.json, or a path you pass with --data.
• For CTIBench, the datasets package and network access to Hugging Face. The benchmark directory includes enterprise-attack.json and mobile-attack.json.

Use a disposable checkout when you want a clean working tree. Several scripts write *_results.json files under benchmarks/.

Prepare the shell

Run the benchmarks against the checkout source and disable background enrichment so ingestion stays deterministic.

export PYTHONPATH=src
export ZETTELFORGE_ENRICHMENT_ENABLED=false
python3 - <<'PY'
import zettelforge
print(zettelforge.__version__)
PY

The version check prints 2.8.0 for the checkout verified in this run. A configured LLM provider can log provider-specific warnings before the version.

Run the CTI retrieval benchmark

python3 benchmarks/cti_retrieval_benchmark.py

This command runs two ingestion strategies over 8 embedded CTI reports and 20 queries. The 2026-08-23 verification run produced these results on this host:

Strategy	Notes	Accuracy	p50 latency
full_session	8	75.0%	60 ms
chunked_800	8	75.0%	57 ms

Accuracy by category stayed unchanged across both strategies:

Category	Accuracy
Attribution	100.0%
Multi-hop	100.0%
CVE linkage	75.0%
Temporal	66.7%
Tool attribution	40.0%

The script verdict for this run was: same accuracy, chunked mode is faster. Treat that as a local benchmark result, not a release decision by itself.

LOCOMO status: in progress

LOCOMO measures conversational memory, not CTI recall. Use the sample command to confirm the harness, dataset path, ingestion path, and keyword judge before you run a larger sample. That quick path does not complete in ZettelForge 2.8.0. The command python3 benchmarks/locomo_benchmark.py --samples 1 --judge keyword loads 10 conversations, 5,882 dialogue turns, and 5 QA pairs, then aborts after 25 sessions. The prompt-injection guard rejects a dataset passage as role_rewrite.

Keep the guard enabled. Do not weaken production ingestion controls to run a third-party benchmark. The benchmark harness needs an upstream, dataset-scoped way to handle this corpus before this path can be documented as working.

RAGAS status: in progress

RAGAS measures retrieval quality. This sample command uses the same LOCOMO data path and evaluates one QA pair per category. In ZettelForge 2.8.0, python3 benchmarks/ragas_benchmark.py --samples 1 aborts after 25 sessions on the same role_rewrite guard finding as LOCOMO. No v2.8.0 RAGAS score ships from this verification run.

Run a CTIBench ATE smoke test

CTIBench ATE tests ATT&CK Technique Extraction. Use a one-sample run first because the adapter loads the Hugging Face dataset and pre-populates ATT&CK technique notes.

python3 benchmarks/ctibench_benchmark.py --task ate --samples 1

The 2026-08-23 smoke test loaded 1 CTI-ATE sample, loaded 293 main ATT&CK techniques from the enterprise and mobile matrices, ingested 292 technique entries, and reported F1: 0.118 with 259 ms p50 latency. Hugging Face logged an unauthenticated-request warning; set HF_TOKEN when you need higher download limits.

In-progress benchmark paths

MemPalace comparison is not verified in this run. The local Python environment has chromadb, but it does not have the mempalace package. Treat the MemPalace numbers in benchmarks/BENCHMARK_REPORT.md as historical until you install that dependency and rerun the comparison.

MemoryAgentBench does not complete under the v2.8.0 defaults. A limit-one AR run loads the Hugging Face dataset, then the prompt-injection guard rejects its first context as instruction_override. Treat the MemoryAgentBench numbers in benchmarks/BENCHMARK_REPORT.md as historical until the harness handles the benchmark corpus without weakening production ingestion controls.

Historical references

The source checkout also carries benchmarks/BENCHMARK_REPORT.md, which records older April 2026 baselines and a June 2026 performance session. Use those numbers to understand methodology and regression history. Use fresh local runs for current performance claims.

Suite	Historical result in source report	Current status in this how-to
CTI retrieval	75.0% accuracy in earlier snapshots	Verified on v2.8.0 at 75.0%
LOCOMO	Historical full-suite baselines retained	In progress: v2.8.0 guard blocks the dataset
RAGAS	Historical LOCOMO retrieval-quality baselines retained	In progress: v2.8.0 guard blocks the dataset
CTIBench ATE	Historical v2.2.0 F1 baseline retained	Smoke-tested only
MemPalace	Historical comparison retained	In progress
MemoryAgentBench	Historical AR/CR results retained	In progress: v2.8.0 guard blocks the dataset

What changes your results

Factor	Effect
ZETTELFORGE_ENRICHMENT_ENABLED	Set it to false before benchmarking so background LLM enrichment does not change ingest timing.
LLM provider availability	LOCOMO answer synthesis logs warnings when the configured provider is unavailable.
Embedding model	This run used fastembed with nomic-ai/nomic-embed-text-v1.5-Q at 768 dimensions.
Reranking policy	Current source enables cross-encoder reranking with 8 candidates and 256 document characters.
Dataset handling	LOCOMO, RAGAS, and MemoryAgentBench currently stop on prompt-injection guard findings before scoring.
Hardware and thread settings	Latency varies by CPU, ONNX thread settings, cache state, and background load.