Skip to content

RFC-012: LiteLLM Unified Provider

RFC-012 records why ZettelForge uses one optional LiteLLM adapter for cloud-hosted generation instead of maintaining a separate adapter for every model provider. Phase 1 shipped in v2.5.0 and remains implemented in v2.8.0. The upstream RFC still records its formal decision as pending.

Use this design record to understand the architecture and its consequences. For current keys, defaults, and provider examples, use the Configuration reference.

Field Value
Author Patrick Roland
Status Implemented — Phase 1; formal RFC decision pending
Created 2026-04-25
Last verified 2026-08-14
Verified release ZettelForge v2.8.0
Related RFC RFC-002 Universal LLM Provider Interface
Related ticket ZF-012

Implementation status

Scope Verified v2.8.0 state
LiteLLM provider class and registry entry Implemented; first shipped in v2.5.0
Optional litellm dependency extra Implemented with a lower bound of litellm>=1.60.0
Per-call provider or model override Not implemented; the public generation function has no provider or model parameter
Provider-specific llm.extra passthrough Not implemented; the client passes the values to the constructor, where this provider ignores unrecognized keyword arguments
Embeddings through LiteLLM Outside this RFC; ZettelForge uses its embedding provider path instead

Context

RFC-002 introduced a provider boundary so extraction, classification, enrichment, and synthesis use one generation function. Dedicated adapters for every hosted model service would duplicate authentication, request conversion, retry behavior, and dependency management.

The design needed to add hosted-model routing without making LiteLLM a dependency of the core ZettelForge installation. It also needed to preserve the existing synchronous generation contract used by the rest of the codebase.

Proposal

Add litellm as a provider name beside local, ollama, and mock. The adapter converts the ZettelForge generation call into a litellm.completion() call. LiteLLM then selects its backend from the configured model identifier.

Keep the SDK in the optional zettelforge[litellm] dependency group. A local-only ZettelForge installation does not install LiteLLM.

Shipped design

Provider boundary

LiteLLMProvider implements the same generate() shape as the other ZettelForge providers. It builds a chat message list and passes the configured model, token limit, temperature, timeout, and retry count to litellm.completion().

The adapter adds api_key only when you configure a non-empty key. In JSON mode, it adds response_format={"type": "json_object"}. It strips surrounding whitespace from the returned message content.

The adapter also sets LiteLLM's suppress_debug_info flag during the completion call and restores the previous value afterward, including when the call raises an exception. This keeps the change scoped to ZettelForge's call.

Registration and dependency loading

ZettelForge imports the provider module and registers the name litellm during provider initialization. The provider module imports the LiteLLM SDK only when generation starts.

This lazy import has an important consequence: available() includes litellm even when the SDK is absent. A missing SDK raises an actionable ImportError when generation starts. Treat the registry list as a list of known provider names, not an installation check.

Defaults have two layers

The provider class uses these constructor fallbacks when a caller supplies no values:

Constructor value Class fallback
Model gpt-4o-mini
API key empty string
Timeout 60.0 seconds
Retry count 2

Normal application calls load the shared llm configuration first. ZettelForge v2.8.0 defines qwen3.5:9b as the shared model default and 180.0 seconds as the shared timeout, then passes those values to the selected provider. Selecting provider: litellm without choosing a compatible model therefore does not activate the provider class's gpt-4o-mini fallback.

Failure and fallback behavior

The adapter logs a failed completion and re-raises the exception. The public ZettelForge client then applies these rules:

  • A missing SDK or provider configuration error reaches the caller. ZettelForge does not hide it with a fallback.
  • Another completion exception invokes the explicitly configured fallback provider.
  • Without an explicit fallback, a failed LiteLLM completion returns an empty string from the public generation function.
  • The historical implicit fallback applies only from local to ollama; it does not add LiteLLM to that chain.

extra does not reach LiteLLM completion

The shared client merges llm.extra into provider constructor arguments. LiteLLMProvider accepts unrecognized constructor arguments for registry compatibility and discards them. Values such as drop_params therefore do not reach litellm.completion() in v2.8.0.

Adding provider-specific completion arguments requires a source change or a future explicit passthrough design. Do not treat llm.extra as a LiteLLM request-parameter channel in this release.

Consequences

Benefits

  • ZettelForge keeps one generation boundary while LiteLLM owns backend-specific request routing.
  • Local and Ollama users avoid the optional cloud-routing dependency.
  • Existing generation call sites require no LiteLLM-specific branches.
  • The adapter preserves the shared logging and fallback boundary.

Trade-offs

  • LiteLLM compatibility and model naming remain upstream concerns.
  • The registry name does not prove the optional SDK is installed.
  • You select one process-wide provider and model configuration; individual calls cannot choose another model.
  • Provider-specific LiteLLM completion options cannot travel through llm.extra in v2.8.0.
  • The interface is synchronous and does not expose streaming through this provider boundary.

Alternatives considered

Maintain one adapter per hosted provider

Rejected. Separate OpenAI-compatible, Anthropic, Bedrock, and Vertex adapters would multiply authentication paths, dependencies, and tests inside ZettelForge.

Make LiteLLM a core dependency

Rejected. The optional dependency keeps the core and local-only installation smaller and avoids installing a cloud-routing SDK for readers who do not use it.

Replace model-based routing with provider-specific configuration fields

Rejected for Phase 1. The implemented adapter sends one model identifier to LiteLLM and leaves backend selection inside that SDK.

Decision

The formal decision remains pending in the v2.8.0 upstream RFC. The shipped code implements the Phase 1 direction: ZettelForge keeps LiteLLM optional, registers one adapter, and routes generation through the shared provider boundary.

That implementation does not constitute approval of the unshipped scope. Per-call model selection, LiteLLM-specific extra passthrough, and embedding support remain outside the implemented decision.