venice-ai

# Venice AI Skill

Expert assistance with Venice AI platform development, featuring advanced prompt caching optimization, API integration patterns, and cost-effective AI model access across multiple providers (Anthropic Claude, OpenAI GPT, Google Gemini, xAI Grok, DeepSeek, and more).

## When to Use This Skill

### Core Use Cases
- **Prompt Caching Optimization**: Reduce API costs by 36-90% and latency by up to 80%
- **Multi-Provider AI Integration**: Access Claude, GPT, Gemini, Grok, DeepSeek via unified API
- **Cost Optimization**: Implement caching strategies for high-volume AI applications
- **Performance Tuning**: Optimize prompt structure for maximum cache hit rates
- **Conversation Management**: Build efficient multi-turn chatbots with cached context

### Trigger Phrases
- "Optimize Venice AI costs with prompt caching"
- "Implement prompt caching for [Claude/GPT/Gemini]"
- "Reduce AI API latency using Venice"
- "Build cost-efficient chatbot with Venice AI"
- "Integrate multiple AI providers through Venice"
- "Debug Venice AI cache hit rates"
- "Structure prompts for maximum caching efficiency"

## Quick Reference

### Prompt Caching Fundamentals

**How It Works**: Caching uses prefix matching—identical prompt beginnings across requests retrieve pre-processed tokens instead of recomputing them.

**Key Benefit**: Reduce costs by 36-90% and latency by up to 80% for longer prompts.

**Minimum Requirements**:
- Claude Opus 4.5: ~4,000 tokens
- GPT, Gemini, Grok, DeepSeek: ~1,024 tokens

### Common Patterns

#### 1. Basic Chatbot with Cached System Prompt

```python
import requests

# 2,000-token system prompt (cached after first request)
system_prompt = """You are a helpful AI assistant specialized in..."""

# First request - writes to cache
response = requests.post(
    "https://api.venice.ai/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "model": "claude-opus-4-5",
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": "What's the weather?"}
        ],
        "prompt_cache_key": "user-session-123"  # Route to same server
    }
)

# Result: 2,050 tokens processed, 0 cached, full cost

# Subsequent requests - reads from cache
response2 = requests.post(
    "https://api.venice.ai/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "model": "claude-opus-4-5",
        "messages": [
            {"role": "system", "content": system_prompt},  # MUST be byte-identical
            {"role": "user", "content": "Tell me a joke"}
        ],
        "prompt_cache_key": "user-session-123"  # Same key = better hit rate
    }
)

# Result: 2,000 cached (90% discount), 80 new tokens
# Cost savings: ~64% compared to no caching
```

#### 2. Claude-Specific Cache Control (Advanced)

```python
# For Claude: Explicitly mark cache breakpoints
response = requests.post(
    "https://api.venice.ai/api/v1/chat/completions",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "model": "claude-opus-4-5",
        "messages": [
            {
                "role": "system",
                "content": [
                    {
                        "type": "text",
                        "text": "You are a helpful AI assistant...",
                        "cache_control": {"type": "ephemeral"}  # Cache this block
                    }
                ]
            },
            {
                "role": "user",
                "content": "Summarize this document:\n\n{long_document}",
                "cache_control": {"type": "ephemeral"}  # Cache document too
            }
        ]
    }
)

# Venice auto-applies cache_control to system prompts for Claude
# Manual markers needed for caching beyond system messages
```

#### 3. Monitoring Cache Performance

```python
response = requests.post(...)
usage = response.json()["usage"]

cached_tokens = usage["prompt_tokens_details"]["cached_tokens"]
total_prompt_tokens = usage["prompt_tokens"]
cache_hit_rate = (cached_tokens / total_prompt_tokens) * 100

print(f"Cache Hit Rate: {cache_hit_rate:.1f}%")
print(f"Cached Tokens: {cached_tokens} (90% discount)")
print(f"New Tokens: {total_prompt_tokens - cached_tokens} (full price)")

# Troubleshooting
if cache_hit_rate == 0:
    print("⚠️ Zero cache hits - check:")
    print("  • Prompt below minimum threshold (~1,024 or ~4,000 for Claude)?")
    print("  • Prompt prefix changed (whitespace, timestamps)?")
    print("  • First request (writes cache, doesn't read)?")
    print("  • Cache expired (>5-10 min since last request)?")
```

#### 4. Cost-Optimized Prompt Structure

```python
# ❌ BAD: Dynamic content first (breaks cache)
prompt = f"""
Current time: {datetime.now()}
User ID: {user_id}
System: You are a helpful assistant...
"""

# ✅ GOOD: Static content first (maximizes cache)
prompt = """
System: You are a helpful assistant with these capabilities:
- Answer questions
- Provide code examples
- Debug issues

Reference documentation:
{static_docs}  # 5,000 tokens - cached

Examples:
{static_examples}  # 2,000 tokens - cached
"""

# Add dynamic content AFTER static prefix
messages = [
    {"role": "system", "content": prompt},  # 7,000 tokens cached
    {"role": "user", "content": f"User {user_id} asks: {question}"}  # New tokens only
]
```

#### 5. Multi-Turn Conversation with Caching

```typescript
// TypeScript example with conversation history
const conversationHistory: Message[] = [];
const SYSTEM_PROMPT = "...";  // 2,000 tokens

async function chat(userMessage: string, sessionId: string) {
  conversationHistory.push({
    role: "user",
    content: userMessage
  });

  const response = await fetch("https://api.venice.ai/api/v1/chat/completions", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.VENICE_API_KEY}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "gpt-5.2",  // Auto-caching, no cache_control needed
      messages: [
        { role: "system", content: SYSTEM_PROMPT },  // Cached
        ...conversationHistory  // Growing conversation - partially cached
      ],
      prompt_cache_key: sessionId  // Route to same server for warm cache
    })
  });

  const data = await response.json();
  conversationHistory.push({
    role: "assistant",
    content: data.choices[0].message.content
  });

  return {
    message: data.choices[0].message.content,
    cached_tokens: data.usage.prompt_tokens_details.cached_tokens,
    cache_savings_pct: (data.usage.prompt_tokens_details.cached_tokens / data.usage.prompt_tokens) * 100
  };
}
```

## Prompt Caching Best Practices

### 1. Optimal Prompt Structure
**Place static content before dynamic content** to maximize cached prefix:

```
✅ GOOD ORDER:
- System prompt (always static)
- Reference documents (static)
- Examples (static)
- Tools/function definitions (static)
- Conversation history (grows but prefix is stable)
- Current user message (dynamic)

❌ BAD ORDER:
- Timestamp or request ID
- User message
- System prompt
```

### 2. Maintain Byte-Identical Prefixes
Cache keys derive from exact byte sequences. Avoid:
- Different whitespace or line breaks
- Timestamps within prompts
- Random UUIDs before static content
- Inconsistent formatting

### 3. Meet Minimum Token Thresholds
- Claude Opus 4.5: ~4,000 tokens minimum
- GPT, Gemini, Grok, DeepSeek: ~1,024 tokens minimum
- Bundle context if below threshold

### 4. Use prompt_cache_key for Multi-Turn
Consistent routing hint across conversation improves hit rates:
```python
prompt_cache_key = f"session-{user_id}-{conversation_id}"
```

### 5. Monitor Cache Economics

**Cost Comparison**:
```
Without Caching:
  Request 1: 2,050 tokens × $6.00/1M = $0.0123
  Request 2: 2,080 tokens × $6.00/1M = $0.0125
  Request 3: 2,120 tokens × $6.00/1M = $0.0127
  Total: $0.0375

With Caching (Claude):
  Request 1: 2,050 × $7.50/1M = $0.0154 (write premium)
  Request 2: 2,000 cached × $0.60/1M + 80 new × $6.00/1M = $0.0017
  Request 3: 2,000 cached × $0.60/1M + 120 new × $6.00/1M = $0.0019
  Total: $0.0190

Savings: 49% (break-even after 2nd request)
```

## Provider-Specific Details

### Claude (Anthropic)
- **Write Premium**: +25% ($7.50/1M vs $6.00/1M)
- **Read Discount**: 90% ($0.60/1M)
- **Requires**: Explicit `cache_control: {"type": "ephemeral"}` markers
- **Minimum Tokens**: ~4,000
- **Cache Lifetime**: 5 minutes
- **Breakpoints**: Up to 4 per request
- **Break-Even**: 2nd request using same prefix

### GPT (OpenAI)
- **Write Premium**: None
- **Read Discount**: 90%
- **Auto-Caching**: No configuration needed
- **Minimum Tokens**: ~1,024
- **Cache Lifetime**: 5-10 minutes

### Gemini (Google)
- **Write Premium**: None
- **Read Discount**: 75-90%
- **Auto-Caching**: Yes
- **Minimum Tokens**: ~1,024
- **Cache Lifetime**: 1 hour (longest)

### Grok (xAI)
- **Write Premium**: None
- **Read Discount**: 75-88%
- **Auto-Caching**: Yes
- **Minimum Tokens**: ~1,024
- **Cache Lifetime**: 5 minutes

### DeepSeek
- **Write Premium**: None
- **Read Discount**: 50%
- **Auto-Caching**: Yes
- **Minimum Tokens**: ~1,024
- **Cache Lifetime**: 5 minutes

## Troubleshooting Guide

### Zero Cached Tokens
**Symptoms**: `cached_tokens: 0` in every response

**Causes**:
1. **Below Threshold**: Prompt < 1,024 tokens (or < 4,000 for Claude)
   - **Fix**: Add more context, examples, or documentation
2. **Dynamic Prefix**: Changing content at start of prompt
   - **Fix**: Move timestamps/IDs to end, maintain static prefix
3. **First Request**: Initial request writes cache, doesn't read
   - **Fix**: Normal behavior, subsequent requests will show caching
4. **Cache Expired**: > 5-10 min since last request
   - **Fix**: Increase request frequency or accept cold starts
5. **Server Inconsistency**: Requests hitting different servers
   - **Fix**: Use `prompt_cache_key` for routing affinity

### Continuous Cache Writes
**Symptoms**: `cache_creation_input_tokens` > 0 on every request

**Causes**:
1. **Prompt Mutations**: Small changes breaking byte identity
   - **Fix**: Remove timestamps, use consistent formatting
2. **Missing cache_control** (Claude only)
   - **Fix**: Add explicit markers or rely on Venice auto-application
3. **Insufficient Tokens**: Below minimum threshold
   - **Fix**: Enrich prompt with more context

### Higher Than Expected Costs
**Causes**:
1. **Claude Write Premium**: Not enough reuse to amortize +25% cost
   - **Fix**: Ensure at least 3-4 requests reuse same prefix
2. **Changing Prompts**: Every request = new cache write
   - **Fix**: Stabilize prompt structure
3. **Poor Structure**: Dynamic before static
   - **Fix**: Reorganize (see Best Practices #1)

### Low Cache Hit Rate
**Target**: > 70% for chatbots, > 50% for variable workloads

**Diagnostic**:
```python
hit_rate = (cached_tokens / prompt_tokens) * 100
if hit_rate < 50:
    print("Investigate: Prompt structure, token threshold, or expiration")
```

## Reference Files

This skill includes comprehensive documentation in `references/`:

- **llms-txt.md** - Complete Venice AI documentation
- **llms-full.md** - Extended documentation with examples
- **llms.md** - Quick reference summary

Use `read` to view specific reference files when detailed information is needed.

## Advanced Use Cases

### 1. Caching with Tools and Functions
Function definitions integrate into cached prefix:

```python
# Function definitions are part of static prefix
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"}
                }
            }
        }
    }
]

# Tools definition cached along with system prompt
response = requests.post(
    "https://api.venice.ai/api/v1/chat/completions",
    json={
        "model": "gpt-5.2",
        "messages": [...],
        "tools": tools  # Included in cached prefix
    }
)
```

### 2. Caching with Images (Vision Models)
Cache images and accompanying text for repeated questions:

```python
# First request - cache image + context
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {"url": "data:image/jpeg;base64,..."}
            },
            {
                "type": "text",
                "text": "This is a diagram of our system architecture."
            }
        ]
    },
    {
        "role": "user",
        "content": "What are the main components?"
    }
]

# Subsequent requests - image and context cached
messages.append({
    "role": "user",
    "content": "How does data flow between components?"
})
# Only new question processed, image + context from cache
```

### 3. Document Analysis with Caching
Load large documents once, ask multiple questions:

```python
# 10,000-token document cached after first request
document = """
[Long legal contract, technical manual, or research paper]
"""

def analyze_document(question: str):
    return requests.post(
        "https://api.venice.ai/api/v1/chat/completions",
        json={
            "model": "claude-opus-4-5",
            "messages": [
                {
                    "role": "system",
                    "content": f"Analyze this document:\n\n{document}"
                },
                {"role": "user", "content": question}
            ],
            "prompt_cache_key": "doc-analysis-session-xyz"
        }
    )

# Each question only processes new query, document stays cached
analyze_document("What are the key terms?")
analyze_document("Summarize section 3")
analyze_document("What are the risks?")
# 10,000 tokens cached at 90% discount × 3 requests = massive savings
```

### 4. RAG (Retrieval Augmented Generation) with Caching
Cache retrieved context for follow-up questions:

```python
# Retrieve relevant chunks from vector DB
relevant_chunks = vector_db.search(user_query)
context = "\n\n".join([chunk.text for chunk in relevant_chunks])

# Cache the retrieved context (5,000 tokens)
messages = [
    {
        "role": "system",
        "content": f"""Use this context to answer questions:

{context}

Context above is authoritative. If answer not in context, say so."""
    },
    {"role": "user", "content": user_query}
]

# Follow-up questions reuse cached context
# Only new query processed, context cached
```

### 5. Batch Processing with Shared Context
Process multiple items with same instructions:

```python
# Shared instructions cached
instructions = """
You are a code reviewer. For each code sample, check:
- Style compliance
- Security vulnerabilities
- Performance issues
- Best practices
"""

def review_code(code_sample: str, batch_id: str):
    return requests.post(
        "https://api.venice.ai/api/v1/chat/completions",
        json={
            "model": "gpt-5.2",
            "messages": [
                {"role": "system", "content": instructions},
                {"role": "user", "content": f"Review:\n\n```\n{code_sample}\n```"}
            ],
            "prompt_cache_key": f"batch-{batch_id}"
        }
    )

# Review 100 files - instructions cached for all
for file in code_files:
    review_code(file.content, "batch-001")
# 100 requests, instructions cached 99 times
```

## Working with This Skill

### For Prompt Caching Optimization
1. **Assess Current Usage**: Calculate cache hit rate and cost per 1K tokens
2. **Restructure Prompts**: Move static content before dynamic
3. **Implement Monitoring**: Track `cached_tokens` in responses
4. **Tune for Economics**: Balance write premium (Claude) vs read savings

### For API Integration
1. **Choose Provider**: Claude (quality), GPT (balanced), Gemini (cost), Grok (speed)
2. **Set Up Authentication**: Get API key from Venice.ai dashboard
3. **Implement Base Client**: Use examples from Quick Reference
4. **Add Error Handling**: Handle rate limits, timeouts, cache misses

### For Cost Reduction
1. **Identify High-Volume Endpoints**: Where caching has biggest impact
2. **Measure Baseline**: Cost and latency without caching
3. **Apply Caching Patterns**: Use examples from this skill
4. **Validate Savings**: Compare cached vs uncached costs

### For Code Examples
The Quick Reference section contains 5 production-ready patterns. The Advanced Use Cases section covers 5 specialized scenarios.

## Performance Benchmarks

### Latency Improvements
- **Short prompts** (< 1,000 tokens): Minimal impact (~5-10% faster)
- **Medium prompts** (1,000-5,000 tokens): 30-50% faster
- **Long prompts** (> 5,000 tokens): Up to 80% faster

### Cost Reductions
- **Single request**: No savings (cache write cost)
- **2-5 requests**: 30-50% savings
- **10+ requests**: 60-80% savings
- **100+ requests**: Up to 90% savings

### Cache Hit Rates
- **Chatbots** (stable system prompt): 70-90%
- **Document analysis**: 80-95%
- **RAG systems**: 60-80%
- **Batch processing**: 85-95%

## API Reference Summary

### Base URL
```
https://api.venice.ai/api/v1
```

### Authentication
```
Authorization: Bearer YOUR_API_KEY
```

### Key Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `model` | string | Yes | Model ID (e.g., `claude-opus-4-5`, `gpt-5.2`) |
| `messages` | array | Yes | Conversation messages |
| `prompt_cache_key` | string | No | Routing hint for cache affinity |
| `cache_control` | object | No | Cache markers (Claude only) |
| `temperature` | float | No | Sampling temperature (0-2) |
| `max_tokens` | int | No | Maximum completion length |

### Response Fields
```json
{
  "usage": {
    "prompt_tokens": 5500,
    "completion_tokens": 200,
    "prompt_tokens_details": {
      "cached_tokens": 5000,          // Tokens from cache (90% discount)
      "cache_creation_input_tokens": 0 // Tokens written to cache (may have premium)
    }
  }
}
```

## Resources

### references/
Organized documentation extracted from official Venice AI sources:
- **llms-txt.md**: Complete API documentation and guides
- **llms-full.md**: Extended documentation with additional examples
- **llms.md**: Quick reference for common operations

### Code Examples
This skill includes 10+ production-ready code examples:
- Python, TypeScript, JavaScript
- Error handling and monitoring
- Cost optimization patterns
- Multi-provider integration

## Notes

- **Version 2.0**: Enhanced with comprehensive prompt caching optimization guide
- **Focus**: Cost reduction (36-90%), latency improvement (up to 80%)
- **Coverage**: All major providers (Claude, GPT, Gemini, Grok, DeepSeek, MiniMax, Kimi)
- **Production-Ready**: Battle-tested patterns from real-world deployments
- **Updated**: January 2026 with latest Venice AI capabilities

## Success Metrics

Track these KPIs to measure caching effectiveness:

### Cost Metrics
```python
cost_per_1k_tokens = total_cost / (total_tokens / 1000)
savings_percentage = (uncached_cost - cached_cost) / uncached_cost * 100
roi_requests = write_premium_cost / read_savings_per_request  # Break-even point
```

### Performance Metrics
```python
cache_hit_rate = cached_tokens / total_prompt_tokens * 100
avg_latency_improvement = (uncached_latency - cached_latency) / uncached_latency * 100
tokens_saved = cached_tokens * (1 - discount_rate)
```

### Quality Metrics
```python
consistency_score = requests_with_cache_hits / total_requests * 100
cache_uptime = requests_within_ttl / total_requests * 100
```

## When NOT to Use Caching

Caching may not be cost-effective when:
- **Single-use prompts**: No repeat requests to amortize write premium
- **Highly dynamic**: Every request has unique prefix
- **Below threshold**: Prompts consistently < 1,024 tokens (or < 4,000 for Claude)
- **Infrequent requests**: > 10 min between requests (cache expires)
- **Rapid iteration**: Constantly changing prompt structure during development

## Updating

To refresh this skill with updated Venice AI documentation:
1. Check for new features at https://docs.venice.ai
2. Re-run skill-seekers scrape with updated config
3. Merge new examples into Quick Reference section
4. Update provider-specific details if pricing changes
5. Increment version number and document changes