Skip to content

RFC-002: Universal LLM Provider Interface

RFC-002 separates ZettelForge's LLM-dependent workflows from specific model backends. The provider infrastructure shipped in v2.3.0 and remains present in v2.8.0. RFC-012 later replaced the proposed provider-specific cloud adapters with the optional LiteLLM provider; per-call provider selection did not ship.

Use this design record to understand the decision and its consequences. For current setup instructions, use Configuration reference. To add a provider, use Build an extension.

Field Value
Author Patrick Roland
Status Accepted — provider infrastructure implemented; later phases superseded or unimplemented
Created 2026-04-16
Last verified 2026-08-08
Verified release ZettelForge v2.8.0
Related RFCs RFC-001 Conversational Entity Extractor · RFC-012 LiteLLM Unified Provider

Context

ZettelForge originally selected between its local GGUF backend and Ollama inside one client module. That design tied extraction, classification, enrichment, and synthesis to backend-specific configuration and failure handling. Adding another backend required changes in the shared client path.

The design needed to preserve the public generation call while giving each backend one implementation boundary. It also needed to keep the cloud-routing SDK optional so the core package remained usable without it.

Decision drivers

  • Keep LLM call sites independent of the selected backend.
  • Preserve the existing public generation call.
  • Keep the cloud-routing dependency optional.
  • Let third-party packages add providers without changing ZettelForge source.
  • Surface permanent configuration failures instead of hiding them behind fallback behavior.
  • Preserve the existing local-to-Ollama recoverable fallback path.

Proposal

Route generation through three layers:

  1. The public zettelforge.llm_client.generate() function reads the active configuration.
  2. A registry resolves a provider name to one cached provider instance.
  3. The provider implements the LLMProvider protocol and performs generation.

The protocol uses typing.Protocol and carries the runtime_checkable marker. Callers can opt into isinstance() checks, but the registry does not validate protocol conformance during registration. A provider exposes a short name and a synchronous generate() method; it does not inherit from a ZettelForge base class.

The registry owns registration, lookup, instance caching, and reset behavior. It protects registration and first-time construction with a lock. Unknown names and duplicate registrations raise ValueError.

Shipped result

Phase 1 shipped in commit f67db7d and release v2.3.0. The same boundary is present in v2.8.0.

Component Verified v2.8.0 behavior
llm_providers.base Defines the runtime-checkable LLMProvider protocol and LLMProviderConfigurationError.
llm_providers.registry Registers provider classes by name and creates one cached instance per provider name.
Built-in providers Registers local, ollama, and mock with the core package.
Optional LiteLLM provider Registers litellm only when its SDK imports successfully.
Entry-point discovery Loads the zettelforge.llm_providers entry-point group during package import. A failed third-party load is logged at debug level and does not stop core startup.
Public LLM client Resolves the configured provider, delegates generation, applies fallback rules, and exposes reload() for configuration changes.

Failure and reload behavior

The public client treats missing SDKs, invalid provider names, and LLMProviderConfigurationError as configuration failures and raises them. It does not replace a permanent configuration problem with output from a different backend.

For other provider exceptions, the client tries the configured fallback. When you leave the fallback empty and select local, it retains the historical local to ollama fallback. If both recoverable paths fail, generation returns an empty string.

The registry caches constructed providers. After you change environment variables or configuration at runtime, llm_client.reload() clears cached configuration and provider instances. It preserves provider registrations.

Phase status

Phase Original scope Verified outcome
1 Provider protocol, registry, built-ins, configuration plumbing, and extension discovery Shipped in v2.3.0; verified in v2.8.0
2 Dedicated OpenAI-compatible provider Superseded by RFC-012
3 Dedicated Anthropic provider Superseded by RFC-012
4 Provider configuration and third-party extension documentation Delivered through the current reference and extension guide
5 Per-call provider override Not implemented in v2.8.0

RFC-012 chose one optional litellm adapter instead of separate cloud-provider files. The v2.8.0 public generate() signature has no provider parameter. Each call resolves the process-wide configuration; reload() lets the process adopt a configuration change, but one call cannot select its own provider.

Consequences

Benefits

  • Extraction, update, classification, enrichment, evolution, and synthesis code use one generation boundary.
  • The core install does not require LiteLLM or a cloud-provider SDK.
  • Tests use the deterministic mock provider without a model service.
  • Third-party packages integrate through package metadata instead of modifying the repository.
  • Configuration failures remain visible at the point of use.

Trade-offs

  • Provider instances are process-wide singletons by name, and constructor options after the first lookup are ignored until reset. You call reload() before expecting runtime configuration changes to take effect.
  • Entry-point load failures appear only in debug logs, so plugin diagnostics require debug logging.
  • The protocol is synchronous and has no streaming method.
  • You cannot route individual calls to different providers through generate() in v2.8.0.

Alternatives considered

Make LiteLLM a core dependency

Rejected for the core install. ZettelForge keeps LiteLLM behind the litellm optional dependency so offline and local-only users do not install its dependency tree.

Use the OpenAI SDK as the universal client

Rejected. It would make one vendor SDK part of the core boundary and would not provide the same multi-provider coverage as the optional LiteLLM adapter.

Require an abstract base class

Rejected. A runtime-checkable protocol provides the required interface without forcing third-party providers to inherit from a ZettelForge class.

Create one configuration section per provider

Rejected. One llm section with a provider discriminator keeps provider selection in one place and avoids adding a top-level section for every adapter.

Decision

Accepted on 2026-04-16. ZettelForge keeps the Phase 1 provider boundary. RFC-012 supersedes the dedicated cloud adapters proposed in Phases 2 and 3. Phase 5 remains unimplemented in v2.8.0 and is not current behavior.