Sigma schema reference¶

Use this page to look up the ZettelForge Sigma parser, schema dispatch, entity mapping, ingest API, and command-line interface. The reference covers ZettelForge 2.8.0, released under the MIT license.

Modules: zettelforge.sigma, zettelforge.sigma.parser, zettelforge.sigma.entities, zettelforge.sigma.tags, zettelforge.sigma.ingest

In progress

Parsing, validation, entity mapping, persistence, and dry-run ingestion work in ZettelForge 2.8.0. Three mappings remain incomplete: filter bodies do not populate SigmaRule.detection_body, tlp.* tags do not populate SigmaRule.tlp, and sigma_format_version remains unset. See Current limitations.

Prerequisites¶

You need ZettelForge 2.8.0 and Python 3.10 or later. Use Configure Sigma ingestion for a task-oriented import procedure. Read Detection rules schema for the shared Sigma and YARA model.

Public exports¶

zettelforge.sigma exports these names:

Name	Kind	Purpose
`SigmaRule`	dataclass	Sigma-specific `DetectionRule` subtype
`SigmaParseError`	exception	Reports YAML, file, and size errors
`SigmaValidationError`	exception	Reports JSON Schema failures
`parse_yaml`	function	Parses and validates YAML text
`parse_file`	function	Reads, parses, and validates one file
`validate`	function	Validates an existing rule dictionary
`from_rule_dict`	function	Builds a `SigmaRule` and relation dictionaries without schema validation
`rule_to_entities`	alias	Points to the same function as `from_rule_dict`
`resolve_sigma_tag`	function	Resolves supported tag namespaces to typed entities
`ingest_rule`	function	Persists one rule and its graph relations
`ingest_rules_dir`	function	Walks and ingests a rule directory

Processing layers¶

Layer	Module	Responsibility
Schema	`sigma.parser` / `sigma.schemas`	Load YAML and validate it against a vendored schema
Entity	`sigma.entities` / `sigma.tags`	Map a validated dictionary to `SigmaRule` and relation dictionaries
Ingest	`sigma.ingest`	Store the rule as a memory note and write knowledge-graph edges

Vendored schemas¶

ZettelForge packages three SigmaHQ JSON schemas in src/zettelforge/sigma/schemas/. The parser loads them through importlib.resources, caches each loaded schema in _SCHEMA_CACHE, and requires no separate schema download.

File	Schema title	Required top-level fields	Selected when
`sigma-detection-rule-schema.json`	Sigma rule specification V2.0.0 (2024-08-08)	`title`, `logsource`, `detection`	Neither `correlation` nor `filter` is present
`sigma-correlation-rules-schema.json`	Sigma Meta rule specification V2.1.0 (2025-07-25)	`title`, `correlation`	`correlation` is present
`sigma-filters-schema.json`	Sigma Global Filter specification V2.0.0 (2024-08-08)	`title`, `logsource`, `filter`	`filter` is present and `correlation` is absent

All three schemas declare JSON Schema draft 2020-12. Dispatch checks correlation first, then filter, then falls back to the detection schema. A dictionary containing both keys therefore selects the correlation schema.

Detection rule schema¶

Top-level properties¶

Property	Type	Required	Description
`title`	string, max 256	yes	Brief detection title
`id`	UUID string	no	Globally unique rule identifier
`name`	string, max 256	no	Name used by correlation rule references
`related`	array of objects	no	Relationships to other rules
`taxonomy`	string, max 256	no	Taxonomy identifier
`status`	enum string	no	Rule lifecycle state
`description`	string, max 65,535	no	Detailed description
`references`	array of strings	no	External references
`author`	string	no	Rule author
`date`	date string	no	Creation date
`modified`	date string	no	Last modification date
`tags`	array of strings	no	Namespaced tags
`level`	enum string	no	Severity level
`logsource`	object	yes	Product, service, category, and definition fields
`detection`	object	yes	Selections plus required `condition` field
`falsepositives`	array of strings	no	Known false-positive scenarios
`fields`	array of strings	no	Output field names
`license`	string	no	License for the rule content, separate from the ZettelForge software license
`scope`	array	no	Scope values defined by the vendored Sigma schema

`status` values¶

Value	Meaning
`stable`	Used in multiple environments for a long period without obvious false positives
`test`	Used on a limited set of test systems without obvious false positives
`experimental`	New and not tested outside lab environments
`deprecated`	Replaced by or covered by another rule linked through `related`
`unsupported`	Not usable in its current form

`level` values¶

Value	Meaning
`informational`	Enrichment-only match that does not trigger a case or alert
`low`	Notable event that rarely represents an incident on its own
`medium`	Relevant event that needs regular manual review
`high`	Relevant event that triggers an internal alert and prompt review
`critical`	Incident-level event with near-certain significance

`logsource` properties¶

Property	Type	Description
`product`	string	Product such as `windows`
`service`	string	Product subset such as `security` or `sshd`
`category`	string	Product group such as `process_creation`
`definition`	string	Free-text source definition

`related[].type` values¶

Value	ZettelForge relation
`obsolete`	`superseded_by`
`derived`	`related_to`
`merged`	`related_to`
`renamed`	`related_to`
`similar`	`related_to`

Each related item requires id and type. Only obsolete produces superseded_by; every other supported value produces related_to.

Correlation schema¶

The correlation schema requires title and correlation. The correlation object requires type, rules, and timespan.

Property	Type	Required	Description
`correlation.type`	enum string	yes	Correlation calculation
`correlation.rules`	array of strings	yes	Referenced rule names or UUIDs
`correlation.timespan`	string, max 10	yes	Lowercase number-plus-unit window such as `15m` or `1h`
`correlation.group-by`	array of strings	conditional	Fields that define separate event scopes
`correlation.condition`	object	conditional	Count comparison and, for value calculations, the target field
`correlation.aliases`	object	no	Field-name aliases applied to referenced rules

Supported correlation.type values are:

event_count
temporal
temporal_ordered
value_count
value_sum
value_avg
value_percentile

The schema also accepts optional correlation metadata such as id, status, description, author, references, date, modified, taxonomy, falsepositives, level, and generate.

Filter schema¶

The filter schema requires title, logsource, and filter. The filter object requires all three fields below.

Property	Type	Description
`filter.rules`	array of strings	Rule names that receive the filter
`filter.selection`	array or object	Filter values or field/value pairs
`filter.condition`	string	Relationship between the filter selections

The filter schema also accepts id, description, date, modified, and taxonomy.

Parser API¶

`parse_yaml`¶

Signature: parse_yaml(text: str) -> dict[str, Any]

parse_yaml loads YAML with yaml.safe_load, converts YAML date and datetime objects back to ISO-8601 strings, selects a schema, and validates the result. It returns the validated dictionary.

It raises:

SigmaParseError for invalid YAML or a top-level value that is not a mapping.
SigmaValidationError when the selected JSON Schema rejects the mapping.

`parse_file`¶

Signature: parse_file(path: str | Path) -> dict[str, Any]

parse_file reads one UTF-8 rule file and delegates to parse_yaml. It includes the path in parse and validation errors.

The parser enforces MAX_RULE_FILE_BYTES = 1_048_576 and raises SigmaParseError before YAML parsing when a file exceeds the limit. It also raises SigmaParseError when it cannot stat or read the path.

`validate`¶

Signature: validate(rule: dict[str, Any]) -> ValidationResult

validate returns a ValidationResult instead of raising. The result contains:

Field	Type	Meaning
`valid`	`bool`	`True` when the selected schema accepts the dictionary
`errors`	`list[str]`	One path-prefixed message per validation failure

bool(result) returns result.valid. A dictionary with no title or logsource and with level: catastrophic returns these errors:

<root>: 'title' is a required property
<root>: 'logsource' is a required property
level: 'catastrophic' is not valid under any of the given schemas

Error types¶

SigmaParseError and SigmaValidationError both extend ValueError.

Exception	Raised for
`SigmaParseError`	Invalid YAML, non-mapping YAML, file stat/read failure, or an oversized file
`SigmaValidationError`	A parsed mapping that fails the selected JSON Schema

Entity model¶

`DetectionRule`¶

SigmaRule extends zettelforge.detection.base.DetectionRule.

Field	Type	Description
`rule_id`	`str`	Upstream UUID or deterministic content-hash identifier
`title`	`str`	Rule title
`source_format`	`str`	`sigma` for Sigma rules
`content_sha256`	`str`	SHA-256 of the canonical, key-sorted YAML form
`description`	`str \\| None`	Rule description
`author`	`str \\| None`	Rule author
`date`	`str \\| None`	Creation date
`modified`	`str \\| None`	Modification date
`references`	`list[str]`	External references
`tags`	`list[str]`	Raw rule tags
`level`	`str \\| None`	Rule severity
`status`	`str \\| None`	Rule lifecycle state
`tlp`	`str \\| None`	Inherited TLP field; not populated by Sigma mapping in 2.8.0
`license`	`str \\| None`	Rule-content license
`source_repo`	`str \\| None`	Upstream repository
`source_path`	`str \\| None`	Path inside the upstream repository
`extra`	`dict[str, Any]`	Extension values

`SigmaRule`¶

Field	Type	Default or behavior
`logsource_product`	`str \\| None`	Copied from `logsource.product`
`logsource_service`	`str \\| None`	Copied from `logsource.service`
`logsource_category`	`str \\| None`	Copied from `logsource.category`
`rule_level`	`str \\| None`	Raw `level` value
`rule_status`	`str \\| None`	Raw `status` value
`sigma_format_version`	`str \\| None`	Remains `None` in 2.8.0
`detection_body`	`str \\| None`	YAML for `detection` or `correlation`; remains `None` for filters
`rule_type`	`str`	`detection`, `correlation`, or `filter`
`fields`	`list[str]`	Copied from `fields`, default `[]`
`falsepositives`	`list[str]`	Copied from `falsepositives`, default `[]`

When a rule has no id, ZettelForge sets rule_id to sigma_ plus the first 16 characters of the canonical content hash. The same rule content therefore produces the same fallback identifier.

Entity conversion¶

Signature: from_rule_dict(rule_dict: dict[str, Any]) -> tuple[SigmaRule, list[dict[str, Any]]]

rule_to_entities is the same function object as from_rule_dict. The function returns a populated SigmaRule plus a list of relation dictionaries. It does not validate its input; pass dictionaries through validate, parse_yaml, or parse_file first.

Each relation dictionary contains:

Key	Value
`from_type`	`SigmaRule`
`from_value`	Source rule identifier
`rel`	ZettelForge relation name
`to_type`	Target entity type
`to_value`	Target entity value
`properties`	Optional relation metadata

Emitted relations¶

Relation	Target type	Emitted when
`applies_to`	`LogSource`	A `product`, `service`, or `category` logsource facet exists
`tagged_with`	`SigmaTag`	A non-empty raw tag exists
`detects`	`AttackPattern`	An `attack.t*` technique or sub-technique tag resolves
`references_cve`	`Vulnerability`	A `cve.*` tag resolves
`attributed_to`	`IntrusionSet`	An `attack.g*` group tag resolves
`attributed_to`	`Malware`	An `attack.s*` software tag resolves
`superseded_by`	`SigmaRule`	`related[].type` is `obsolete`
`related_to`	`SigmaRule`	`related[].type` is another supported value

Logsource targets use facet:value, such as product:windows. ZettelForge always keeps a supported or unsupported tag as tagged_with; it adds a typed edge only when resolve_sigma_tag returns a target.

Tag resolution¶

Signature: resolve_sigma_tag(tag: str) -> tuple[str, str] | None

Input	Return value
`attack.t1059`	`('AttackPattern', 'T1059')`
`attack.t1059.001`	`('AttackPattern', 'T1059.001')`
`attack.g0007`	`('IntrusionSet', 'G0007')`
`attack.s0154`	`('Malware', 'S0154')`
`cve.2021-44228`	`('Vulnerability', 'CVE-2021-44228')`
`cve.2024.3094`	`('Vulnerability', 'CVE-2024-3094')`
`tlp.amber`	`None`
`detection.emerging`	`None`
`attack.initial-access`	`None`

CVE resolution accepts a dot or hyphen between the year and identifier and returns the canonical hyphenated value.

Ingest API¶

`ingest_rule`¶

Signature: ingest_rule(rule, mm, *, domain='detection', source_ref=None, sync=True) -> tuple[note, relations]

Parameter	Accepted value	Default	Behavior
`rule`	dictionary, YAML string, path string, or `Path`	required	Strings and paths are parsed and validated; dictionaries are used as-is
`mm`	`MemoryManager`-compatible object	required	Receives the note and knowledge-graph edges
`domain`	string	`detection`	Stored note domain
`source_ref`	string or `None`	`None`	Overrides the generated source reference
`sync`	boolean	`True`	Controls inline enrichment in `MemoryManager.remember`

The generated source reference is sigma:<rule_id>:<first-12-content-hash-characters>. If the store already contains that reference, the function returns the existing note and newly computed relations without persisting duplicates.

For a new rule, the function stores the YAML and a one-line summary as source_type='sigma_rule', then attempts to persist every relation. It raises ValueError when mm is None and propagates parser and validation errors from string and path inputs. Dictionary input bypasses schema validation.

`ingest_rules_dir`¶

Signature: ingest_rules_dir(path, mm, *, glob='**/*.yml', domain='detection', bulk=False, flush_timeout=None) -> tuple[int, int]

The function returns (ingested, skipped). It raises FileNotFoundError when the root path does not exist. It matches both .yml and .yaml, logs per-file failures, increments skipped, and continues the walk.

When bulk=True, it calls each ingest_rule with sync=False and calls mm.flush(timeout=flush_timeout) once after the walk.

Directory security controls¶

Control	Behavior
File size	`parse_file` rejects files larger than 1,048,576 bytes
YAML loading	`yaml.safe_load` prevents Python object construction
Symlinks	`ingest_rules_dir` skips symlinked files
Root containment	`ingest_rules_dir` skips resolved paths outside the requested root

Knowledge-graph edge metadata¶

Before persistence, ingest_rule adds these properties unless the relation already supplies them:

Property	Value	Purpose
`edge_type`	`detection`	Distinguishes detection edges from causal and heuristic edges
`source`	`sigma_ingest`	Records ingest provenance

If one edge write raises an exception, ZettelForge logs sigma_edge_persist_failed and continues with the next relation.

CLI¶

The verified dry-run invocation against the v2.8.0 release fixture is:

python3 -m zettelforge.sigma.ingest --dry-run tests/fixtures/sigma/tagged_example.yml

Argument	Default	Description
`path`	required	One `.yml` or `.yaml` file, or a directory
`--dry-run`	off	Parses, validates, and maps without persistence
`--domain`	`detection`	Memory domain for persisted notes
`--glob`	`*/.yml`	Directory glob; ZettelForge also checks the corresponding `.yaml` glob

A successful dry run writes:

OK  tests/fixtures/sigma/tagged_example.yml  id=7e3d88a2-bfaa-4f52-9e0b-2bbbdd0d4ea1  type=detection  tags=5  edges=11

Dry-run summary: 1/1 parsed, 0 failed.

ZettelForge 2.8.0 also exposes python3 -m zettelforge.sigma.cli. The ingest module invocation emits a non-fatal runpy warning on stderr because zettelforge.sigma.__init__ imports ingest before Python executes it as a module. The command still exits successfully; use the cli module to avoid that warning.

The CLI does not invoke the LLM rule explainer in 2.8.0.

Quick reference example¶

This self-contained example parses a rule and inspects the entity relations without writing data:

from zettelforge.sigma import from_rule_dict, parse_yaml

yaml_text = """title: Command shell start
logsource:
  product: windows
  category: process_creation
detection:
  selection:
    Image: cmd.exe
  condition: selection
tags:
  - attack.t1059.003
"""

rule_dict = parse_yaml(yaml_text)
entity, relations = from_rule_dict(rule_dict)
print(entity.rule_id, entity.rule_type)
print([(edge["rel"], edge["to_value"]) for edge in relations])

Output:

sigma_158aaf05e376149d detection
[('applies_to', 'product:windows'), ('applies_to', 'category:process_creation'), ('tagged_with', 'attack.t1059.003'), ('detects', 'T1059.003')]

Current limitations¶

These ZettelForge 2.8.0 behaviors are in progress:

from_rule_dict validates filter rules and sets rule_type='filter', but it leaves detection_body=None instead of preserving the filter object.
tlp.* tags remain available through tags and tagged_with relations, but they do not populate the inherited tlp field.
sigma_format_version remains None for parsed detection, correlation, and filter rules.
from_rule_dict and dictionary input to ingest_rule bypass schema validation; validate dictionaries before calling either API.
The CLI parses, validates, maps, and persists rules, but it does not run the LLM rule explainer.