YARA schema reference¶

Use this page to look up the ZettelForge YARA parser, CCCS metadata validation, entity mapping, ingest API, and command-line interface. The reference covers ZettelForge 2.8.0, released under the MIT license.

Modules: zettelforge.yara, zettelforge.yara.parser, zettelforge.yara.cccs_metadata, zettelforge.yara.entities, zettelforge.yara.tags, zettelforge.yara.ingest

In progress

Parsing, core metadata validation, entity mapping, persistence, and dry-run ingestion work in ZettelForge 2.8.0. Modifier flags, several optional CCCS validators, reference mapping, strict live-ingest exit status, dry-run symlink protection, and CLI explainer integration remain incomplete. See Current limitations.

Prerequisites¶

You need ZettelForge 2.8.0 and Python 3.10 or later. Use Configure YARA ingestion for a task-oriented import procedure. Read Detection rules schema for the shared Sigma and YARA model.

Public exports¶

zettelforge.yara exports these names:

Name	Kind	Purpose
`YaraRule`	dataclass	YARA-specific `DetectionRule` subtype
`YaraParseError`	exception	Reports file stat and size errors
`YaraValidationError`	exception	Reserved validation exception; the 2.8.0 APIs return `ValidationResult` instead of raising it
`parse_yara`	function	Parses YARA source text
`parse_file`	function	Reads and parses one rule file
`validate_metadata`	function	Validates flattened metadata at a selected tier
`rule_to_entities`	function	Builds a `YaraRule` and relation dictionaries
`resolve_yara_tag`	function	Resolves supported inline tags to typed entities
`ingest_rule`	function	Persists the first rule from one input
`ingest_rules_dir`	function	Walks and ingests a rule directory

zettelforge.yara.parser.parse_text is a legacy alias for parse_yara. It is available from the parser submodule but is not a package-level export.

Processing layers¶

Layer	Module	Responsibility
Parser	`yara.parser`	Wrap Plyara and normalize parsed dictionaries
Metadata	`yara.cccs_metadata` / `yara.schemas`	Load the CCCS metadata definitions and apply the implemented validators
Entity	`yara.entities` / `yara.tags`	Map a parsed dictionary to `YaraRule` and relation dictionaries
Ingest	`yara.ingest`	Store each rule as a memory note and write knowledge-graph edges

Vendored CCCS schemas¶

ZettelForge packages two CCCS YARA metadata files in src/zettelforge/yara/schemas/:

File	Purpose
`CCCS_YARA.yml`	Declares metadata fields, formats, uniqueness, optionality, and validator names
`CCCS_YARA_values.yml`	Declares allowed status, sharing, category, malware-type, actor-type, and hash patterns

The YAML files come from the Cyber Centre's CCCS-Yara project and retain its MIT license. ZettelForge does not vendor the upstream validator_functions.py; cccs_metadata.py implements the validation logic locally. The module loads both YAML files at import time, so a missing or unreadable file prevents the module from importing.

Metadata validation¶

Signature: validate_metadata(rule_meta: dict[str, Any], tier: Tier = 'warn') -> ValidationResult

ValidationResult contains three fields:

Field	Type	Meaning
`accepted`	`bool`	Whether the selected tier accepts the metadata
`warnings`	`list[str]`	Non-blocking missing-field and value findings
`errors`	`list[str]`	Blocking strict-tier findings

Validation tiers¶

Tier	Behavior
`warn`	Records missing required fields and invalid values as warnings; always sets `accepted=True`
`strict`	Records missing or invalid required fields as errors and sets `accepted=False`; invalid optional fields remain warnings
`non_cccs`	Skips every check and returns `accepted=True` with empty warning and error lists

Strict-tier ingest drops a rule when its entity is not marked strict. Warn-tier ingest keeps the rule and records the findings in YaraRule.extra.

Fields required by ZettelForge strict mode¶

ZettelForge combines two CCCS optionality classes into one strict gate. It requires every optional: No field and every optional: Optional field because the local validator does not generate missing values.

Field	CCCS optionality	Implemented format or values
`id`	`Optional`	Base62 text with at least 16 characters
`fingerprint`	`Optional`	40–64 hexadecimal characters
`version`	`Optional`	Two numeric components, such as `1.0`
`modified`	`Optional`	`YYYY-MM-DD` text
`status`	`No`	`TESTING`, `RELEASED`, or `DEPRECATED`
`sharing`	`No`	One of the supported TLP strings below
`source`	`No`	Any non-empty value; uppercase is a convention, not a validation rule
`author`	`No`	Whole value matches `[A-Za-z0-9_.@+-]+`
`description`	`No`	Non-empty after trimming
`category`	`No`	`INFO`, `EXPLOIT`, `TECHNIQUE`, `TOOL`, or `MALWARE`

Supported sharing values are:

TLP:CLEAR and TLP:CLEAR//COMMERCIAL
TLP:WHITE and TLP:WHITE//COMMERCIAL
TLP:GREEN and TLP:GREEN//COMMERCIAL
TLP:AMBER and TLP:AMBER//COMMERCIAL
TLP:AMBER+STRICT and TLP:AMBER+STRICT//COMMERCIAL

TLP:WHITE remains accepted alongside TLP:CLEAR for compatibility with existing rules.

Optional fields with implemented validation¶

Field	Accepted value
`date`	`YYYY-MM-DD` text
`malware_type`	One of the values in Malware type values
`actor_type`	`APT`, `CRIMEWARE`, or `FIN`
`hash`	32-, 40-, or 64-character hexadecimal value
`mitre_att`	One `TA####`, `T####`, `T####.###`, `M####`, `G####`, or `S####` identifier

An invalid optional value produces a warning in both warn and strict mode; it does not make strict validation fail. The entity converter separately splits comma- and semicolon-delimited mitre_att text, but the metadata validator accepts only one identifier per value.

Declared fields without implemented validation¶

The vendored schema declares validators for these optional fields, but ZettelForge 2.8.0 accepts their values without applying those validators:

Field	Format declared by the vendored schema
`score`	Integer percentage
`minimum_yara`	`x.x` version
`actor`	Actor name
`mitre_group`	ATT&CK group alias
`report`	Linked report
`reference`	Free-form external reference
category child metadata	Category-specific uppercase value

The schema also declares unvalidated fields such as tags, vol_script, al_*, credit, and original_*. Unknown metadata keys pass through without a warning.

Malware type values¶

The implemented malware_type validator accepts:

ADWARE, APT, BACKDOOR, BANKER, BOOTKIT, BOT, BROWSER-HIJACKER, BRUTEFORCER, CLICKFRAUD, CRYPTOMINER, DDOS, DOWNLOADER, DROPPER, EXPLOITKIT, FAKEAV, HACKTOOL, INFOSTEALER, KEYLOGGER, LOADER, OBFUSCATOR, POS, PROXY, RAT, RANSOMWARE, REVERSE-PROXY, ROOTKIT, SCANNER, SCAREWARE, SPAMMER, TROJAN, VIRUS, WIPER, WEBSHELL, and WORM.

Parser API¶

`parse_yara`¶

Signature: parse_yara(text: str) -> list[dict[str, Any]]

parse_yara parses source text into one dictionary per rule. It protects a module-level plyara.Plyara instance with _PARSER_LOCK, clears the parser before and after every call, flattens Plyara's metadata list into meta, and carves raw_rule from the source line range.

The normalizer always adds these keys:

Key	Type	Meaning
`meta`	`dict`	Flattened metadata; the last duplicate key wins
`tags`	`list[str]`	Inline tags, or an empty list
`raw_rule`	`str`	Source text between Plyara's start and stop lines

Plyara supplies the remaining rule-dependent keys, including rule_name, metadata, strings, condition_terms, raw_condition, raw_strings, raw_meta, start_line, stop_line, and scopes. ZettelForge also copies parser-level imports into a rule when Plyara omits them from that rule.

Plyara syntax failures surface as ValueError with a plyara parse error message.

`parse_file`¶

Signature: parse_file(path: str | Path) -> list[dict[str, Any]]

parse_file stats the path, rejects files larger than MAX_RULE_FILE_BYTES = 1_048_576, reads UTF-8 text, and delegates to parse_yara.

Failure	Exception
Path cannot be statted	`YaraParseError`
File exceeds 1,048,576 bytes	`YaraParseError`
Plyara rejects the text	`ValueError`
UTF-8 read fails after a successful stat	The underlying read or decode exception

YaraParseError extends ValueError.

Entity model¶

`DetectionRule`¶

YaraRule extends zettelforge.detection.base.DetectionRule.

Field	Type	YARA mapping
`rule_id`	`str`	CCCS `id`, or `yara_` plus the first 16 content-hash characters
`title`	`str`	Parsed rule name
`source_format`	`str`	`yara`
`content_sha256`	`str`	SHA-256 of `raw_rule`, or a stable fallback projection
`description`	`str \\| None`	CCCS `description`
`author`	`str \\| None`	CCCS `author`
`date`	`str \\| None`	CCCS `date`
`modified`	`str \\| None`	CCCS `modified`
`references`	`list[str]`	One CCCS `report` value when present; `reference` is not mapped
`tags`	`list[str]`	Raw inline YARA tags
`level`	`str \\| None`	Remains `None`
`status`	`str \\| None`	Lowercase CCCS `status`
`tlp`	`str \\| None`	CCCS `sharing` value
`license`	`str \\| None`	Remains `None`
`source_repo`	`str \\| None`	Remains `None`
`source_path`	`str \\| None`	Remains `None`; directory ingest writes the path into the note body
`extra`	`dict[str, Any]`	Validation and parser details described below

`YaraRule`¶

Field	Type	Mapping
`cccs_id`	`str \\| None`	CCCS `id`
`fingerprint`	`str \\| None`	CCCS `fingerprint`
`category`	`str \\| None`	CCCS `category`
`technique_tag`	`str \\| None`	CCCS `technique`
`cccs_version`	`str \\| None`	String form of CCCS `version`
`hash_of_sample`	`list[str]`	One or more CCCS `hash` values
`rule_name`	`str \\| None`	Parsed rule name
`is_private`	`bool`	Remains `False` in 2.8.0 even when Plyara reports a private scope
`is_global`	`bool`	Remains `False` in 2.8.0 even when Plyara reports a global scope
`imports`	`list[str]`	Parsed YARA module imports
`condition`	`str \\| None`	Trimmed raw condition

extra always contains cccs_compliant, cccs_warnings, cccs_errors, condition_terms, and source_line_range. It also contains source and malware_type when present. If metadata supplies actor_type without an actor name, extra contains actor_type and the converter emits no actor relation.

Entity conversion¶

Signature: rule_to_entities(rule: dict[str, Any], *, tier: str = 'warn') -> tuple[YaraRule, list[dict[str, Any]]]

The function validates rule['meta'] at the requested tier and returns a YaraRule plus relation dictionaries. It returns the entity even when strict validation fails; the ingest layer decides whether to drop it.

zettelforge.yara.entities.from_rule_dict(rule) is a compatibility wrapper that calls rule_to_entities at the default warn tier. It is not a package-level export.

Each relation dictionary contains:

Key	Value
`from_type`	`YaraRule`
`from_value`	Source rule identifier
`rel`	ZettelForge relation name
`to_type`	Target entity type
`to_value`	Target entity identifier
`properties`	Relation metadata

Emitted relations¶

Relation	Target type	Emitted when
`detects`	`AttackPattern`	CCCS `mitre_att` contains a token, including `TA`, `T`, `M`, `G`, or `S` identifiers
`detects`	`AttackPattern`	An inline tag resolves as a technique
`attributed_to`	`ThreatActor`	CCCS `actor` contains a name
`tagged_with`	`YaraTag`	CCCS `technique` contains a value
`tagged_with`	`YaraTag`	An inline tag resolves as a category or free-form tag
`references_cve`	`Vulnerability`	An inline tag resolves as a CVE

The CCCS mitre_att mapper uppercases and splits list, comma-delimited, and semicolon-delimited values. It maps every token to AttackPattern; it does not distinguish group, software, or tactic identifiers.

Tag resolution¶

Signature: resolve_yara_tag(tag: str) -> tuple[str, dict[str, str]]

Input pattern	Returned entity and properties
`T1218`, `attack.T1218`, or `attack_T1218`	`AttackPattern` with `technique_id='T1218'`
`attack.T1059.003`	`AttackPattern` with `technique_id='T1059.003'`
`CVE_2021_44228` or `CVE-2021-44228`	`Vulnerability` with canonical `cve_id`
Known category token	`YaraTag` with namespace `category` and an uppercase name
Any other value	`YaraTag` with namespace `freeform` and the trimmed input

Known category tokens are APT, CRIME, CRIMEWARE, EXPL, HKTL, MAL, MALWARE, PUA, RAT, RANSOM, RANSOMWARE, SUSP, VULN, and WEBSHELL. Matching is case-insensitive.

Each inline tag emits one relation. A technique or CVE tag emits its typed relation instead of an additional raw-tag relation.

Ingest API¶

`ingest_rule`¶

Signature: ingest_rule(rule_source, mm=None, *, domain='detection', tier='warn', sync=True) -> tuple[MemoryNote | None, list[dict]]

Parameter	Accepted value	Default	Behavior
`rule_source`	Dictionary, YARA text, path string, or `Path`	required	Normalizes the input and ingests only the first parsed rule
`mm`	`MemoryManager`-compatible object	`None`	Raises `ValueError` when omitted or `None`
`domain`	string	`detection`	Stored note domain
`tier`	`strict`, `warn`, or `non_cccs`	`warn`	Controls CCCS validation
`sync`	boolean	`True`	Controls inline enrichment in `MemoryManager.remember`

Strict rejection returns (None, relations). A new accepted rule becomes a memory note with source_type='yara'. The generated source reference is yara:<rule_id>:<first-12-content-hash-characters>.

If the store already contains that source reference, the function returns the existing note and newly computed relations without persisting a duplicate.

`ingest_rules_dir`¶

Signature: ingest_rules_dir(path, mm=None, *, glob='**/*.yar', tier='warn', domain='detection', bulk=False, flush_timeout=None) -> dict[str, Any]

The return dictionary has ingested, skipped, and errors fields. A missing root returns {'ingested': 0, 'skipped': 0, 'errors': ['path does not exist: ...']} instead of raising.

The default glob checks both .yar and .yara files. A custom glob replaces that pair. The function records parse and per-rule failures in errors, counts strict rejections and idempotent hits as skipped, and continues the walk.

When bulk=True, the function calls each rule ingest with sync=False and calls mm.flush(timeout=flush_timeout) once after the walk when mm exposes a flush method. A false flush result produces a warning log and does not add a return error.

Directory security controls¶

Control	Behavior
File size	`parse_file` rejects files larger than 1,048,576 bytes
Symlinks	`ingest_rules_dir` skips symlinked files
Root containment	`ingest_rules_dir` skips resolved paths outside the requested root

Knowledge-graph edge persistence¶

Before persistence, ingest adds these properties unless the relation already supplies them:

Property	Value
`edge_type`	`detection`
`source`	`yara_ingest`

If one edge write raises an exception, ZettelForge logs yara_edge_persist_failed and continues with the next relation.

CLI¶

The verified dry-run invocation against the v2.8.0 release fixture is:

python3 -m zettelforge.yara.ingest tests/fixtures/yara/technique_loader.yar --dry-run

Argument	Default	Description
`path`	required	One `.yar` file or a directory
`--tier`	`warn`	`strict`, `warn`, or `non_cccs`
`--dry-run`	off	Parses, validates, and maps without persistence
`--domain`	`detection`	Memory domain for persisted notes
`--json`	off	Emits JSON instead of the human-readable summary

The successful command writes this stdout and exits 0:

[warn] MemoryModule  (technique_loader.yar)  relations=2  mitre=T1218
    warn: missing required CCCS field: id
    warn: missing required CCCS field: fingerprint
    warn: missing required CCCS field: modified

The ingest module invocation emits a non-fatal runpy warning on stderr because zettelforge.yara.__init__ imports ingest before Python executes it as a module. The command still runs.

Dry-run mode returns exit 1 for parse failures and strict-tier rejection. Live mode returns exit 1 when the result contains errors or the input path is missing. In 2.8.0, a strict-tier live rejection increments skipped but leaves errors empty, so the command prints ingested=0 skipped=1 errors=0 and incorrectly exits 0.

The CLI does not invoke the LLM rule explainer in 2.8.0.

Quick reference example¶

This self-contained example parses a rule and inspects its entity relations without writing data:

from zettelforge.yara import parse_yara, rule_to_entities

yara_text = """rule CommandShell : APT T1218 CVE_2021_44228 {
    strings:
        $cmd = "cmd.exe"
    condition:
        $cmd
}
"""

rule_dict = parse_yara(yara_text)[0]
entity, relations = rule_to_entities(rule_dict, tier="non_cccs")
print(entity.rule_id, entity.rule_name, entity.extra["cccs_compliant"])
print([(edge["rel"], edge["to_value"]) for edge in relations])

Output:

yara_bb08e39bbce9a1f8 CommandShell non_cccs
[('tagged_with', 'APT'), ('detects', 'T1218'), ('references_cve', 'CVE-2021-44228')]

Current limitations¶

These ZettelForge 2.8.0 behaviors are in progress:

Plyara records private and global modifiers in scopes, but entity conversion leaves YaraRule.is_private and YaraRule.is_global false.
The optional CCCS validators listed under Declared fields without implemented validation do not run.
The metadata validator accepts one mitre_att identifier, while entity conversion also splits compound values. The converter maps every resulting identifier type to AttackPattern.
YaraRule.references maps report but ignores reference.
Strict-tier rejection in live CLI mode exits 0 when no separate ingest error occurs.
CLI dry-run directory traversal follows matching symlinked rule files; the ingest_rules_dir persistence path skips them.
YaraValidationError is exported but no YARA 2.8.0 public function raises it.
The CLI parses, validates, maps, and persists rules, but it does not run the LLM rule explainer.