LiteLLM Provider

The default LLM provider in Augments Agents, using litellm to access 100+ language models through a unified API.

Quick Start

from augments.adk.agents import Agent
from augments.adk.llms import LiteLLM, LLMConfig
 
agent = Agent(
    name="Assistant",
    system_prompt="You are helpful.",
    llm=LiteLLM(),
)

If no llm is specified on an Agent, the Runner uses LiteLLM() by default.

Authentication

LiteLLM reads API keys from environment variables:

export ANTHROPIC_API_KEY=sk-ant-...
export OPENAI_API_KEY=sk-...
export GEMINI_API_KEY=...

Or pass explicitly:

llm = LiteLLM(api_key="sk-...", base_url="https://custom-endpoint.com")

Package Structure

All litellm-specific code lives in augments.adk/llms/litellm/:

llms/
├── llm.py               # LLM abstract base class (provider-agnostic)
├── llm_config.py         # LLMConfig @dataclass (provider-agnostic)
├── llm_usage.py          # Token usage tracking (provider-agnostic)
└── litellm/              # LiteLLM-specific implementation
    ├── litellm_model.py              # LiteLLM class
    ├── litellm_param_types.py        # *Param TypedDicts for wire-format dicts
    ├── litellm_prompt_caching_resolver.py # Prompt caching resolver
    ├── litellm_reasoning_resolver.py # Reasoning parameter resolver
    └── litellm_provider.py           # Provider detection

Provider-agnostic types (LLM, LLMConfig, LLMUsage) stay at the llms/ root. This separation means the native provider implementations (e.g., AnthropicLLM, GeminiLLM, OpenAIResponsesLLM) inherit no litellm-specific code.

Configuration

LLMConfig is a provider-agnostic @dataclass. The LiteLLM class maps its fields to litellm parameter names via explicit named arguments — no dict spreading.

from augments.adk.llms import LLMConfig
 
config = LLMConfig(
    temperature=0.7,
    max_output_tokens=2000,
    top_p=0.9,
)

Parameter Mapping

Resilience

config = LLMConfig(
    num_retries=3,                                      # Retry on 429, 500, timeouts
    fallbacks=["gpt-4o-mini", "claude-sonnet-4-20250514"],  # Try these on total failure
)

Prompt Caching

Prompt caching reduces costs by reusing previously computed token representations. Each provider has a different mechanism.

Anthropic

Anthropic uses explicit cache_control breakpoints injected into messages. litellm's AnthropicCacheControlHook handles injection automatically.

from augments.adk.types.caching import AnthropicPromptCaching, CacheTTL
 
config = LLMConfig(
    prompt_caching=AnthropicPromptCaching(
        ttl=CacheTTL.ONE_HOUR,        # 1h cache (2x write, 0.1x read)
        max_cache_breakpoints=2,
        apply_to_system_prompt=True,
    ),
)

How it works in litellm:

1. resolve_cache_params() builds cache_control_injection_points (e.g., system message + last user message)
2. litellm's AnthropicCacheControlHook injects {"type": "ephemeral"} at those points
3. Anthropic's API caches content at the breakpoints

Cost savings:

• Cache write: 1.25x base input token price
• Cache read (hit): 0.1x base input token price
• 5-minute TTL is free; 1-hour TTL costs 2x

Limitations:

• litellm ignores the ttl field — it always uses Anthropic's default (5 minutes). The native AnthropicLLM provider honours a TTL via AnthropicConfig.cache_control_ttl ("5m" / "1h").
• Minimum tokens: varies by model (2048–4096)

Google Gemini

Gemini uses reference-based caching: create a CachedContent resource via Google's API, then reference it by ID.

from augments.adk.types.caching import GeminiPromptCaching
 
config = LLMConfig(
    prompt_caching=GeminiPromptCaching(
        cached_content_id="cachedContents/abc123",
        cache_ttl=3600,  # 1 hour in seconds
    ),
)

How it works in litellm:

1. resolve_cache_params() returns the cached_content ID
2. litellm passes it to Gemini's generateContent API as cachedContent
3. Gemini reuses the pre-cached content

Note: Gemini also supports inline caching via message-level cache_control blocks, but that mechanism is handled by litellm's Vertex AI transformation layer separately — not through the injection points (which are Anthropic-only).

OpenAI

OpenAI caches automatically for prompts >= 1024 tokens. Optional hints improve hit rates.

from augments.adk.types.caching import OpenAIPromptCaching
 
config = LLMConfig(
    prompt_caching=OpenAIPromptCaching(
        prompt_cache_key="my-app-stable",   # Routing hint
        prompt_cache_retention="24h",   # Cache duration
    ),
)

Cost savings: Cached tokens are 50% cheaper (0.5x).

Structured Output

LiteLLM implements a three-tier strategy for structured output:

1. JSON Schema mode — Model natively constrains output to match the schema (best reliability)
2. JSON Object mode — Model outputs valid JSON, client validates against schema
3. No JSON support — Prompt-based + client-side validation

from pydantic import BaseModel
 
class Analysis(BaseModel):
    sentiment: str
    confidence: float
 
agent = Agent(
    name="Analyzer",
    output_schema=Analysis,
    llm=LiteLLM(),
)

Wire-Format Types

All known-shape parameter dicts in litellm_model.py use *Param TypedDicts (defined in litellm_param_types.py) instead of dict[str, Any]. These are plain dicts at runtime — no conversion cost — but give static type checkers visibility into key names and value types.

dict[str, Any] is intentionally kept for extra_params / extra_kwargs (genuinely dynamic provider overrides) and JSON Schema fields (inherently unstructured).

Streaming

result = Runner.run(agent, "Tell me a story", stream=True)
 
async for event in result.stream_events():
    if event.type == "raw_response_event":
        if event.data.type == "content_delta":
            print(event.data.content_delta, end="", flush=True)

Usage tracking in streaming mode requires LLMConfig.include_usage=True (default), which sets stream_options={"include_usage": True} on the litellm call.

Provider Detection

Utilities for detecting which provider backs a model:

from augments.adk.llms.litellm.litellm_provider import detect_provider, is_anthropic
 
detect_provider("claude-sonnet-4-20250514")  # "anthropic"
detect_provider("gpt-4o")                   # "openai"
detect_provider("gemini/gemini-2.5-flash")  # "vertex_ai"
 
is_anthropic("claude-sonnet-4-20250514")    # True

Supported Models

LiteLLM supports 100+ models. Common examples:

See litellm supported models for the full list.