Skip to content

Maintain LanceDB indexes

Inspect and optimize LanceDB after a large ingest or when vector writes slow down. You first measure every table without changing it, stop active writers, then run the verified optimization path.

Prerequisites

  • You run ZettelForge 2.8.0 on the host that owns the data directory.
  • You use the same Python environment that imports zettelforge and lancedb.
  • You have a current backup of the data directory.
  • You can stop every process that writes notes during optimization. Readers can remain active.

The default data directory is ~/.amem. If you set AMEM_DATA_DIR, use that directory instead.

1. Inspect the tables

Set DATA_DIR to the directory that contains vectordb/, then run a dry inspection:

DATA_DIR="${AMEM_DATA_DIR:-$HOME/.amem}"
python -m zettelforge.scripts.compact_lance --data-dir "$DATA_DIR" --dry-run

The command discovers every *.lance directory under vectordb/ and prints one JSON report. Check these fields for each table:

  • before_fragments — data fragments visible before maintenance.
  • before_bytes — bytes below that table's .lance directory.
  • row_count — rows LanceDB can read.
  • error — null when inspection succeeds.

ZettelForge creates one table per memory domain. A CTI domain uses notes_cti; other domains use names such as notes_general and notes_detection. There is no single shared notes table.

2. Stop writers

Stop agents and services that call remember() before a mutating run. The script requires --force as an explicit acknowledgement and refuses mutation without it.

Keep the maintenance window in place until the command exits. The script does not request LanceDB's exclusive-access deletion option, so readers can continue, but concurrent writers add avoidable risk while table files change.

3. Optimize every table

Run the current LanceDB API path:

python -m zettelforge.scripts.compact_lance \
  --data-dir "$DATA_DIR" \
  --all \
  --mode optimize \
  --force

Use --table notes_cti instead of --all when you want to target only that table. The command writes progress to stderr and a JSON report to stdout. Treat exit code 0 as success, 1 as a table error, and 2 as an argument or environment error.

The default compact mode is in progress in ZettelForge 2.8.0

A clean 2.8.0 installation does not install the pylance module that the deprecated compact_files() path requests. Running the script's default compact mode exits with code 1. Use --mode optimize, which completes with the standard installation.

Optimization does not guarantee that a tiny table's fragment count or byte size decreases. LanceDB can create maintenance metadata or new fragments during the operation. Judge the result by a null error, exit code 0, and your before-and-after workload measurements.

4. Inspect again and resume writers

Run the dry inspection a second time:

python -m zettelforge.scripts.compact_lance --data-dir "$DATA_DIR" --dry-run

Confirm that every table reports error: null, then restart your writers. If a table reports an error, keep its writers stopped, preserve the JSON report, and repair the environment before retrying.

Configure automatic version cleanup

ZettelForge also runs a background version-cleanup thread for each active LanceDB table. The threads begin on the first vector write, not when MemoryStore is constructed. They call LanceDB's cleanup_old_versions() method; they do not replace the offline optimization workflow above.

The verified defaults run cleanup every 60 minutes and make versions older than 3,600 seconds eligible for pruning:

lance:
  cleanup_interval_minutes: 60
  cleanup_older_than_seconds: 3600

Set cleanup_interval_minutes: 0 to disable automatic cleanup. Restart the ZettelForge process after editing config.yaml so the running process loads the new values. A failed cleanup cycle logs a warning and leaves the background thread available for its next interval.