ai-gateway

# ai-gateway

Unified LLM proxy with rate limiting, caching, logging, and automatic fallback. Drop-in replacement for direct provider calls.

## When to use

- Calling multiple LLM providers from one endpoint
- Adding response caching to reduce API costs
- Implementing automatic fallback when a provider fails
- Enforcing per-team or per-application rate limits
- Centralizing LLM usage logs and cost tracking
- A/B testing different models or providers
- Any application calling OpenAI, Anthropic, or Cohere APIs

## Prerequisites

1. A running ai-gateway instance (local or Docker)
2. A gateway API key (`gw_` prefix)
3. At least one provider configured in the gateway admin

## Quick Start

```bash
# Start the gateway
gateway start

# Create a gateway API key
gateway key create --label myapp

# Point your OpenAI client at the gateway
# (change the base URL, keep your application logic unchanged)
```

## Usage in Code

### Node.js with OpenAI SDK

```typescript
import OpenAI from 'openai'

const client = new OpenAI({
  apiKey: 'gw_your_gateway_key',
  baseURL: 'http://localhost:4080/v1',
})

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Hello' }],
})
```

### Python with OpenAI SDK

```python
import openai

client = openai.OpenAI(
    api_key="gw_your_gateway_key",
    base_url="http://localhost:4080/v1",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)
```

### Direct HTTP

```bash
curl http://localhost:4080/v1/chat/completions \
  -H "Authorization: Bearer gw_your_gateway_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}]
  }'
```

## Model Routing

The gateway routes requests to providers based on the model name:

| Model pattern | Default provider |
|---|---|
| `gpt-4*` | OpenAI |
| `claude-*` | Anthropic |
| `command-*` | Cohere |
| unknown model | Highest-priority provider that lists it |

Custom routing rules can override defaults. Set them in the admin dashboard or API.

## Caching

Responses are cached automatically when `temperature` is 0 or not set. The cache key is derived from the model, messages, and parameters. On a cache hit, the stored response is returned without calling the provider.

Cache TTL defaults to 1 hour. Configure in settings:

```
GATEWAY_CACHE_TTL=3600   # seconds
GATEWAY_CACHE_ENABLED=1  # 0 to disable
```

## Rate Limiting

Each gateway key has a requests-per-minute limit. The default is 60 rpm (global setting). Override per key:

```bash
gateway key create --label myapp --rate-limit 100
```

On limit exceeded, the gateway returns:
```
HTTP 429 Too Many Requests
Retry-After: 15
```

## Fallback Chains

When a provider fails (timeout, 5xx, or provider-side 429), the gateway automatically tries the next provider in the configured fallback chain. The response includes standard OpenAI format regardless of which provider served it.

Configure fallback in the admin dashboard under Routes.

## CLI Reference

| Command | Description |
|---|---|
| `gateway start` | Start the gateway |
| `gateway key create --label <name>` | Create a gateway API key |
| `gateway key create --label <name> --rate-limit <rpm>` | Create key with custom rate limit |
| `gateway key list` | List all gateway keys |
| `gateway key revoke <id>` | Revoke a key |
| `gateway provider add` | Add a provider interactively |
| `gateway provider list` | List providers |
| `gateway provider health` | Check all provider health |
| `gateway --help` | Show help |
| `gateway --version` | Show version |

## Environment Variables

| Variable | Description | Default |
|---|---|---|
| `GATEWAY_PORT` | Proxy port | 4080 |
| `GATEWAY_ADMIN_PORT` | Admin API + dashboard port | 4081 |
| `GATEWAY_ENCRYPTION_KEY` | 32-byte hex key for provider key encryption | required |
| `GATEWAY_DATA_DIR` | Data directory for SQLite and logs | ~/.ai-gateway |
| `GATEWAY_REDIS_URL` | Redis URL for distributed rate limiting | (none) |
| `GATEWAY_CACHE_ENABLED` | Enable response cache (1 or 0) | 1 |
| `GATEWAY_CACHE_TTL` | Cache TTL in seconds | 3600 |
| `GATEWAY_RATE_LIMIT_RPM` | Default rate limit per key | 60 |
| `GATEWAY_LOG_RETENTION` | Request log retention in days | 30 |
| `GATEWAY_DEV` | Dev mode: verbose logging, no auth (1 or 0) | 0 |

## Troubleshooting

### "invalid gateway key"

Key is wrong, revoked, or missing `gw_` prefix. Check with `gateway key list`.

### Request goes to wrong provider

Check routing rules in the admin dashboard. Model name must match the glob pattern. Default routing uses provider priority order.

### Cache never hits

Ensure `temperature` is 0 or absent. Cache only stores deterministic responses. Check `GATEWAY_CACHE_ENABLED=1`.

### 429 from gateway vs 429 from provider

Gateway 429 means your key hit the gateway rate limit. Provider 429 (wrapped in a 502) means the provider rejected the request. Check the error response body for the source.