RFC-012: LiteLLM Unified Provider for LLM Routing¶

Field	Value
Author	Patrick Roland
Status	Draft — Phase 1 implemented
Created	2026-04-25
Last updated	2026-04-25
ZettelForge version	v2.7.0
Related RFCs	RFC-002 Universal LLM Provider Interface (this supersedes Phases 2–3); RFC-011 Local LLM Backend Selection
Related tickets	ZF-012

Implementation status¶

Phase	Description	Status
Phase 1	LiteLLMProvider class + registry registration + `litellm` extra	Shipped in v2.5.0 (2026-04-25)
Phase 2	Per-call `model=` override via `generate(extra=...)`	Not shipped — depends on RFC-002 Phase 5

The formal decision field in this RFC remains pending, but Phase 1 shipped during the v2.5.0 compliance release. Use provider: litellm in your config.yaml today.

Context¶

ZettelForge needs one registered provider file per LLM backend. RFC-002 proposed openai_compat (Phase 2) and anthropic (Phase 3) as separate provider files, but neither shipped: each requires dedicated auth handling, retry logic, streaming, and tests against a different SDK.

This approach does not scale. A user on AWS Bedrock, Google Vertex, Azure OpenAI, Groq, or any of 100+ available providers cannot use ZettelForge with those backends until a dedicated provider ships.

Proposal¶

Add litellm as a first-class provider name in the registry. One provider file routes to every supported backend via litellm.completion(). Users set provider: litellm in their config and a model name — LiteLLM resolves the correct backend from the model name prefix.

Who benefits¶

Any user of a non-Ollama, non-local backend (OpenAI, Anthropic, Groq, Together AI, etc.).
Users who switch providers frequently — model name is the only config change.
Azure OpenAI, Bedrock, and Vertex users — complex auth is handled by LiteLLM automatically.
The project — no need to ship and maintain openai_compat, anthropic, azure_openai, or bedrock provider files.

Design¶

Architecture¶

LiteLLM is a new top-level provider alongside local, ollama, and mock. It satisfies the same LLMProvider protocol as every other provider. No changes to generate(), the registry interface, or any of the seven callers.

Model name prefix routing:

Model name	Routes to
`gpt-4o`, `gpt-4o-mini`	OpenAI
`claude-sonnet-4-20250514`	Anthropic
`gemini/gemini-2.0-flash`	Google Gemini
`groq/llama-3.3-70b-versatile`	Groq
`together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo`	Together AI
`bedrock/anthropic.claude-3-sonnet-20240229-v1:0`	AWS Bedrock
`vertex_ai/claude-3-sonnet@20240229`	Google Vertex AI
`openrouter/anthropic/claude-3.5-sonnet`	OpenRouter

Provider implementation¶

Shipped as src/zettelforge/llm_providers/litellm_provider.py. Key properties verified from source:

LiteLLMProvider.name = "litellm"
Default model: gpt-4o-mini (when no model is configured)
Retry: delegated to LiteLLM via num_retries kwarg — no manual retry loop
json_mode=True: passes response_format={"type": "json_object"} — works for OpenAI and most OpenAI-compatible endpoints; LiteLLM silently drops it for providers that do not support it
stderr banner suppression: after v2.5.0, the provider scopes litellm.suppress_debug_info=True around each litellm.completion() call and restores the prior value in a finally block. This suppresses LiteLLM's "Provider List" banner that would otherwise appear on stderr approximately 40 times per recall() call when LLM NER is active. The suppression is scoped to one call so it does not affect other code in the same process that uses LiteLLM directly.

Constructor parameters:

Parameter	Type	Default	Description
`model`	`str`	`"gpt-4o-mini"`	Model name. LiteLLM routes based on prefix.
`api_key`	`str`	`""`	API key. Accepts `${ENV_VAR}` references. Leave empty to use standard env vars.
`timeout`	`float`	`60.0`	Request timeout in seconds.
`max_retries`	`int`	`2`	Retries on transient failure (delegated to LiteLLM).

Registry registration¶

__init__.py registers "litellm" conditionally:

try:
    from zettelforge.llm_providers.litellm_provider import LiteLLMProvider
    register("litellm", LiteLLMProvider)
except ImportError:
    pass  # litellm package not installed

The base package never imports litellm unless the SDK is present.

API key strategy¶

Three approaches, all supported:

Config-level key — api_key: ${OPENAI_API_KEY} in config.yaml. The config loader resolves the ${ENV_VAR} reference before the provider sees it. Preferred for single-provider setups.
Environment-level key — Set OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, etc. in the environment. LiteLLM reads these automatically when api_key is empty. Preferred for multi-provider setups.
Per-model env vars — When using multiple providers, set the relevant env var for each and switch models in config.

Config schema¶

llm:
  provider: litellm
  model: gpt-4o                       # LiteLLM routes to correct provider
  api_key: ${OPENAI_API_KEY}          # optional — env vars also work
  temperature: 0.1
  timeout: 60.0
  max_retries: 2

No changes to LLMConfig. The existing fields (provider, model, api_key, temperature, timeout, max_retries, fallback, extra) are sufficient.

Installation¶

pip install zettelforge[litellm]

Requires litellm>=1.60.0. The base pip install zettelforge never pulls in LiteLLM.

Example configurations¶

# OpenAI
llm:
  provider: litellm
  model: gpt-4o
  api_key: ${OPENAI_API_KEY}

# Anthropic
llm:
  provider: litellm
  model: claude-sonnet-4-20250514
  api_key: ${ANTHROPIC_API_KEY}

# Groq (fast inference)
llm:
  provider: litellm
  model: groq/llama-3.3-70b-versatile
  api_key: ${GROQ_API_KEY}

# Google Gemini
llm:
  provider: litellm
  model: gemini/gemini-2.0-flash
  # GOOGLE_API_KEY in env, or api_key: ${GOOGLE_API_KEY}

# AWS Bedrock
llm:
  provider: litellm
  model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
  # AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY in env

# Google Vertex AI
llm:
  provider: litellm
  model: vertex_ai/claude-3-sonnet@20240229
  # GOOGLE_APPLICATION_CREDENTIALS in env

# OpenRouter
llm:
  provider: litellm
  model: openrouter/anthropic/claude-3.5-sonnet
  api_key: ${OPENROUTER_API_KEY}

File changes¶

File	Change
`src/zettelforge/llm_providers/litellm_provider.py`	Created — `LiteLLMProvider` class
`src/zettelforge/llm_providers/__init__.py`	Registers `"litellm"` conditionally
`pyproject.toml`	Adds `litellm = ["litellm>=1.60.0"]` optional extra
`config.default.yaml`	Documents `provider: litellm` with examples
`tests/test_llm_providers.py`	16 unit tests for `LiteLLMProvider` (all pass)

No changes to config.py, llm_client.py, or local_provider.py.

Migration¶

New users: pip install zettelforge[litellm], then set provider: litellm and your model name in config.yaml.

Existing local/Ollama users: No change required. LiteLLM is an additional provider.

Fallback policy: LiteLLM does not have an implicit fallback to local or ollama. If a LiteLLM call fails (missing API key, network error), the error surfaces to the caller. Configure an explicit fallback: ollama in LLMConfig if you want a fallback.

Rollback: Set provider: ollama or provider: local in config.

Alternatives considered¶

Alternative 1 — Ship openai_compat + anthropic + bedrock + vertex separately. Rejected: requires 4+ provider files with separate test suites; each has different auth; 90+ other providers remain unsupported.

Alternative 2 — LiteLLM as a core dependency. Rejected: LiteLLM pulls ~20 transitive dependencies (openai, anthropic, boto3, google-cloud-aiplatform, httpx, etc.); many ZettelForge users never touch cloud providers.

Alternative 3 — Replace openai_compat and anthropic with LiteLLM. Rejected: backward compatibility concern; LiteLLM is heavier than a single-SDK provider.

Alternative 4 — Per-provider extra fields instead of model name routing. Rejected: LiteLLM's model-name prefix routing is the standard pattern; adds config complexity for no benefit.

Open questions¶

Implicit fallback? LiteLLM should not fall back implicitly to local or ollama — failing is the right behavior when a cloud API key is misconfigured. Explicit fallback: ollama remains available.
drop_params / other LiteLLM kwargs via extra? Supported today: extra: { drop_params: true } passes through to litellm.completion().
Embedding support? LiteLLM supports litellm.embedding(). This RFC covers generation only. Follow-up if users request it.
Adding LiteLLM to the local → ollama fallback chain? No. LiteLLM is an external API provider, not a local inference backend.

Decision¶

Field	Value
Decision	Pending formal review — Phase 1 implemented
Date	Pending
Decision maker	Pending
Rationale	Pending