Maintain LanceDB indexes

ZettelForge accumulates fragments in its LanceDB vector store as you ingest notes. Each remember() call writes a new fragment to a domain-specific table. On write-heavy deployments, fragment accumulation degrades insert performance. Two mechanisms address this: an automatic background daemon that prunes old version chains, and an offline compaction script that merges fragments.

How fragmentation happens

LanceDB uses a log-structured format. Each write appends a new fragment to the table instead of rewriting the existing data. Insert latency grows as LanceDB walks a longer list of fragments to service each new write. A shard with 17,000+ fragments (as on the default notes_cti table after sustained use) can produce tail latencies of 55 seconds per insert.

Tables are domain-namespaced. Your ~/.amem/vectordb/ directory contains directories like:

notes_cti.lance/
notes_general.lance/
notes_security_ops.lance/
notes_detection.lance/

Each domain you write to gets its own table. There is no single notes table.

Automatic version cleanup (daemon)

ZettelForge includes a background daemon that prunes old LanceDB version chains automatically. It starts when MemoryStore initializes and runs one daemon thread per active table, waking on a configurable interval.

You do not invoke this daemon manually. Control it with two config keys under lance: in your config.yaml:

lance:
  cleanup_interval_minutes: 60       # how often to run per table; 0 disables
  cleanup_older_than_seconds: 3600   # prune versions older than this

These defaults mean: once per hour, prune any version chain entries older than 1 hour. Set cleanup_interval_minutes: 0 to disable the daemon without restarting ZettelForge.

The daemon is best-effort: any error in a single cleanup cycle is logged at warning level and the thread continues. A maintenance failure never crashes the running process.

Offline compaction (manual)

For shards that have accumulated many fragments before the daemon was active, or after a batch-ingestion event, run the one-shot compaction script:

# Dry-run — inspect fragment counts without touching data
python -m zettelforge.scripts.compact_lance \
    --data-dir ~/.amem \
    --dry-run

Example dry-run output (actual captured output, 2026-06-23):

{
  "data_dir": "/home/rolandpg/.amem",
  "mode": "dry-run",
  "tables": [
    {
      "table": "notes_cti",
      "mode": "dry-run",
      "before_fragments": 17759,
      "before_bytes": 177761640,
      "row_count": 17759
    },
    {
      "table": "notes_general",
      "mode": "dry-run",
      "before_fragments": 995,
      "before_bytes": 8551895,
      "row_count": 995
    }
  ],
  "totals": {
    "before_fragments": 20296,
    "before_bytes": 212401032
  }
}

When you are ready to compact, quiesce any agents writing to ZettelForge, then run with --force:

# Compact all tables — requires --force to acknowledge writers are quiesced
python -m zettelforge.scripts.compact_lance \
    --data-dir ~/.amem \
    --all \
    --force

# Compact one table only
python -m zettelforge.scripts.compact_lance \
    --data-dir ~/.amem \
    --table notes_cti \
    --force

# Full optimize: compact + prune old versions + reindex
python -m zettelforge.scripts.compact_lance \
    --data-dir ~/.amem \
    --all \
    --mode optimize \
    --force

Compaction is safe alongside concurrent readers. Concurrent writers are allowed by LanceDB but not recommended for a one-shot compaction run. Quiesce any agents actively calling remember() before running with --force.

The script exits with code 0 on success, 1 if any table had an error, 2 on argument or environment failures.

Choosing between compact and optimize

Mode	What it does	When to use
compact (default)	Merges fragments into larger files	After batch ingestion; fastest operation
optimize	Compact + prune old version metadata + reindex	After running compact many times; reduces on-disk size further

Run compact first. If disk usage is still high after compaction, follow with optimize.

When to compact

You do not need to compact frequently if the background daemon is running. The daemon handles version chain growth automatically. Run the offline script when:

• You have just completed a large batch ingestion (thousands of notes in one session).
• Fragment count from a dry-run exceeds 10,000 for a single table.
• Insert latency has measurably degraded since the last compaction.

Under typical daily use with the daemon enabled, monthly or quarterly offline compaction is sufficient.

Related

• Tune LanceDB vector search — embedding, retrieval, and reranking settings
• Reference: Configuration — all lance.* config keys
• src/zettelforge/lance_maintenance.py — background daemon source
• src/zettelforge/scripts/compact_lance.py — offline compaction script source