Skip to content

Configure Sigma ingestion

Use this guide when you have Sigma rules and want ZettelForge to store them as memory notes and graph them as detection-rule entities.

File-based Sigma ingestion in ZettelForge 2.8.0 parses YAML, validates each file against the vendored SigmaHQ schemas, maps the rule to a SigmaRule, persists a memory note, and writes detection-rule knowledge graph edges. It does not call the detection-rule explainer during ingest; call the explainer separately after the rule is stored.

CLI symlink refusal is in progress

The 2.8.0 CLI follows matching .yml and .yaml symlinks during directory traversal. Run CLI ingestion only on rule directories you trust. Use the Python ingest_rules_dir() path when you need matching symlinks rejected.

Prerequisites

  • ZettelForge 2.8.0 installed. Start with the Quickstart if you have not installed it yet.
  • A directory of Sigma .yml or .yaml rules.
  • A configured ZettelForge data directory for live ingest. For a disposable trial, set AMEM_DATA_DIR to an empty temporary directory.

Validate a rule set

Dry-run mode is the safest first step. It parses, validates, and maps matching rules, but it does not write memory notes or graph edges. The commands below assume SIGMA_RULE_DIR points to your Sigma rule directory.

python3 -m zettelforge.sigma.ingest "$SIGMA_RULE_DIR" --dry-run

Successful output lists each rule, its resolved ID, rule type, tag count, and edge count:

OK  rules/cloud_example.yml  id=929a690e-bef0-4204-a928-ef5e620d6fcb  type=detection  tags=0  edges=2
OK  rules/tagged_example.yml  id=7e3d88a2-bfaa-4f52-9e0b-2bbbdd0d4ea1  type=detection  tags=5  edges=11

Dry-run summary: 2/2 parsed, 0 failed.

The command exits non-zero when any file fails. Fix FAIL lines before you run live ingest.

Ingest the rules

After dry-run succeeds, ingest the same directory into the detection memory domain:

python3 -m zettelforge.sigma.ingest "$SIGMA_RULE_DIR" --domain detection

Live ingest creates one memory note for each new rule body and persists the emitted knowledge graph edges through the active storage backend. The ingest path is idempotent by source_ref: re-ingesting an unchanged rule returns the existing note instead of creating a duplicate. The content-hash suffix lets a changed body become a new note without losing the upstream rule identifier. Sigma source references use this format:

sigma:<rule_id>:<content_sha256[:12]>

Use the Python API

Use the Python API when you need to validate rules before writing, override the memory domain, or defer enrichment for a large import.

from zettelforge import MemoryManager
from zettelforge.sigma import ingest_rule, ingest_rules_dir, parse_file, validate

mm = MemoryManager()

rule = parse_file("rules/cloud_example.yml")
result = validate(rule)
if not result.valid:
    raise SystemExit(result.errors)

note, relations = ingest_rule(rule, mm, domain="detection")
print(note.id)
print(sorted({rel["rel"] for rel in relations}))

ingested, skipped = ingest_rules_dir(
    "rules",
    mm,
    glob="**/*.yml",
    domain="detection",
    bulk=True,
    flush_timeout=60.0,
)
print(f"ingested={ingested} skipped={skipped}")

ingest_rule() accepts a parsed dict, raw YAML text, a Path, or a string path. ingest_rules_dir() walks recursively, skips invalid files, refuses matching symlinks, and accepts both .yml and .yaml files. A parsed dict bypasses schema validation inside ingest_rule(), so keep the explicit validate() gate shown above.

What ZettelForge writes

Each ingested Sigma rule becomes a MemoryNote with source_type="sigma_rule". The note content contains the normalized YAML plus a one-line summary with the title, level, status, and logsource facets.

The graph edge set depends on the rule fields:

Rule input Edge written
logsource.product, logsource.service, logsource.category applies_to -> LogSource
Any raw Sigma tag tagged_with -> SigmaTag
attack.t* tag detects -> AttackPattern
cve.* tag references_cve -> Vulnerability
attack.g* or attack.s* tag attributed_to -> IntrusionSet or Malware
related[].type: obsolete superseded_by -> SigmaRule
Other related[] entries related_to -> SigmaRule

Raw tagged_with edges are always preserved. When a tag also resolves to a typed entity, ZettelForge writes both the raw provenance edge and the typed edge.

Current limits

  • A single Sigma file is capped at 1 MB.
  • CLI directory traversal follows matching symlinks in 2.8.0. Symlink refusal is in progress for the CLI; use ingest_rules_dir() for the enforced path.
  • ingest_rules_dir() rejects matching symlinks and counts each as skipped.
  • Dry-run mode validates and maps rules only; it does not prove the storage backend can write.
  • The detection-rule explainer is not auto-invoked on ingest in ZettelForge 2.8.0. See Use detection rules and the explainer when you want LLM-generated explanations.
  • ZettelForge 2.8.0 emits a Python RuntimeWarning when invoked as python3 -m zettelforge.sigma.ingest because the package imports the module before runpy executes it. Treat OK/FAIL lines and the process exit code as the ingest result.