Build a Self-Hosted AI Gateway with LiteLLM Proxy

Running AI across your team means juggling multiple provider APIs, API keys scattered across projects, and unpredictable costs. LiteLLM proxy gives you one OpenAI-compatible endpoint that routes to 100+ providers — with virtual keys, rate limits, per-team budgets, and automatic fallbacks.

In this tutorial, you’ll deploy a production-ready LiteLLM proxy with Docker, configure OpenAI and Anthropic backends, set up virtual keys with budgets, and connect it to your tools.

Prerequisites

• Docker Engine 24+ and Docker Compose v2
• OpenAI API key and Anthropic API key
• A host with 2 GB RAM and 10 GB disk (a $10 VPS or your dev machine) [1]
• 30 minutes

Step 1 — Deploy LiteLLM with Docker Compose

Create a project directory and a docker-compose.yml:

# docker-compose.yml
services:
  litellm:
    image: ghcr.io/berriai/litellm:v1.85.0
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml
      - ./litellm.db:/app/litellm.db
    environment:
      - STORE_MODEL_IN_DB=True
      - DATABASE_URL=sqlite:///app/litellm.db
    command:
      - "--config"
      - "/app/config.yaml"
      - "--port"
      - "4000"
      - "--detailed_debug"

Create config.yaml with your providers:

# config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: "latency-based-routing"
  num_retries: 2
  fallbacks:
    gpt-4o: ["claude-sonnet", "gpt-4o-mini"]
    claude-sonnet: ["gpt-4o"]

Set your keys and start the proxy:

export OPENAI_API_KEY="sk-proj-..."
export ANTHROPIC_API_KEY="sk-ant-..."
docker compose up -d

Verify it’s running:

curl -s http://localhost:4000/health | python3 -m json.tool

Expect: {"status": "healthy"}.

Step 2 — Create Virtual Keys with Budgets

Virtual keys let you issue per-user or per-team credentials without exposing your real provider keys. Install the LiteLLM CLI:

pip install litellm

Create a key for your dev team:

litellm --create-key \
  --team_id "dev-team" \
  --team_alias "Dev Team" \
  --models gpt-4o,claude-sonnet \
  --max_budget 50.0 \
  --budget_duration "30d"

The output includes a key value — treat this like a password. Your team uses this key instead of their personal OpenAI key.

Create a second key for CI pipelines with a tighter budget:

litellm --create-key \
  --team_id "ci-pipeline" \
  --team_alias "CI Pipeline" \
  --models gpt-4o-mini \
  --max_budget 10.0 \
  --budget_duration "7d"

All keys and budgets are stored in litellm.db — they survive container restarts.

Step 3 — Configure Rate Limits

Prevent a single team from saturating your upstream quota. Add this to config.yaml:

router_settings:
  routing_strategy: "latency-based-routing"
  num_retries: 2
  fallbacks:
    gpt-4o: ["claude-sonnet", "gpt-4o-mini"]
    claude-sonnet: ["gpt-4o"]
  allowed_rpm: 60
  allowed_tpm: 100000

team_configs:
  - team_id: "dev-team"
    max_budget: 50.0
    rpm: 30
  - team_id: "ci-pipeline"
    max_budget: 10.0
    rpm: 10

Restart the proxy to apply:

docker compose restart

Step 4 — Test from Your Tools

LiteLLM proxy exposes an OpenAI-compatible /chat/completions endpoint. Any tool that supports a custom OpenAI base URL can connect.

Via curl:

curl http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-YOUR-VIRTUAL-KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello, gateway!"}],
    "max_tokens": 50
  }'

Via Python (OpenAI SDK):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="sk-YOUR-VIRTUAL-KEY"
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello, gateway!"}]
)
print(response.choices[0].message.content)

Via Claude Code or Cursor:

Set the environment variable:

# Claude Code
export ANTHROPIC_BASE_URL="http://localhost:4000"

# Cursor → Settings → Models → OpenAI Base URL
# http://localhost:4000
# API Key: sk-YOUR-VIRTUAL-KEY

Step 5 — Verify Fallback Behavior

Test that fallbacks work by passing an invalid model name:

# This should fail and trigger the gpt-4o → claude-sonnet fallback
curl -s http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-YOUR-VIRTUAL-KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Test fallback"}]
  }' | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['choices'][0]['message']['content'][:100])"

Check the proxy logs to confirm fallback fired:

docker compose logs litellm | grep -i fallback

Step 6 — Expose Securely (Optional)

For team access beyond localhost, add a Cloudflare Tunnel or reverse proxy:

# Install cloudflared on the host
cloudflared tunnel create litellm-gateway
cloudflared tunnel route dns litellm-gateway gateway.yourdomain.com

# Run tunnel pointing at the proxy
cloudflared tunnel run --url http://localhost:4000 litellm-gateway

Your team now hits https://gateway.yourdomain.com with their virtual keys.

Verification Checklist

After setup, confirm each of these works:

• curl localhost:4000/health returns healthy
• Chat completion succeeds with a virtual key
• Wrong virtual key returns 401
• Exceeding RPM limit returns 429
• Claude Code connects and routes through the proxy

What You Built

A self-hosted AI gateway with:

• Unified endpoint — one URL for OpenAI, Anthropic, and 100+ providers
• Virtual keys — per-team credentials with isolated budgets
• Automatic fallbacks — Claude takes over when OpenAI errors
• Rate limiting — global and per-team RPM caps
• Cost tracking — every call logged with spend attribution

Source: https://docs.litellm.ai/docs/ and https://github.com/BerriAI/litellm

This runs on your infrastructure, under your control. No external gateway vendor, no per-request markup, no data leaving your network unless it’s going to the model provider.

📖 Related Reads

• NoCode Insider — AI workflow automation with no-code tools, agents, and APIs

Cross-links automatically generated from NiteAgent.

References

• [0] LiteLLM Documentation — https://docs.litellm.ai/docs/
• [1] LiteLLM GitHub Repository — https://github.com/BerriAI/litellm

← Back to all posts