tokenizer-guide

# tokenizer-guide

Reference guide for cl100k_base (the GPT tokenizer) and strategies for reducing token usage in LLM applications.

## What is a token?

A token is the basic unit of text that LLMs process. Tokens are not words, characters, or bytes - they are subword fragments produced by Byte-Pair Encoding (BPE). On average in English text:

- 1 token = approximately 4 characters
- 1,000 tokens = approximately 750 words
- 1 page of text = approximately 500-800 tokens

## cl100k_base encoding

cl100k_base is the tokenizer used by:

- GPT-4 (all variants)
- GPT-4o
- GPT-3.5-turbo
- GPT-4-turbo
- text-embedding-ada-002

It has a vocabulary of ~100,000 tokens (hence "100k"). Spaces are typically merged with the following word to form one token (" hello" is different from "hello").

## How BPE works

1. Start with individual UTF-8 bytes as the initial vocabulary
2. Count the most frequently occurring byte pair in the training corpus
3. Merge that pair into a new single token
4. Repeat until the target vocabulary size is reached

The resulting tokenizer efficiently represents common words as single tokens while splitting rare words into multiple subword tokens.

## Tokenization rules

### Case sensitivity

Uppercase and lowercase letters can map to different tokens. "Python" (capitalized) and "python" (lowercase) are often different token IDs.

### Leading spaces

" word" (with leading space) and "word" (without) are usually different tokens. This is a quirk of how GPT tokenizers are trained on concatenated text.

### Numbers

Individual digits are single tokens. Multi-digit numbers may be split:
- "123" - likely 1 token
- "12345" - may be 2-3 tokens

### Punctuation

Most punctuation marks are single tokens. Some combinations (like "...") may be one token.

### Code and special characters

Code tokenizes differently from prose. Variable names with underscores, camelCase, and symbols often split in non-obvious ways.

## Common token counts by content type

| Content | Approximate tokens |
|---|---|
| Single common word | 1 |
| Single rare/long word | 2-4 |
| One sentence (~15 words) | 15-25 |
| One paragraph (~100 words) | 100-150 |
| System prompt (concise) | 50-150 |
| System prompt (verbose) | 300-800 |
| Short story (1,000 words) | 1,200-1,500 |
| 1,000-row CSV (compact) | 5,000-15,000 |
| Large codebase file | 2,000-10,000 |

## Context windows by model

| Model | Context window | Approximate pages |
|---|---|---|
| gpt-4 | 8,192 tokens | 6 pages |
| gpt-3.5-turbo | 16,385 tokens | 12 pages |
| gpt-4-turbo | 128,000 tokens | 96 pages |
| gpt-4o | 128,000 tokens | 96 pages |
| claude-3-5-sonnet | 200,000 tokens | 150 pages |
| claude-3-opus | 200,000 tokens | 150 pages |
| gemini-1.5-pro | 1,000,000 tokens | 750 pages |

## Reducing token usage

### System prompts

Keep system prompts under 200 tokens when possible. Every request includes the full system prompt - a 300-token reduction at 1,000 requests/day saves 300,000 tokens/day.

Remove redundant instructions: "Always be helpful and professional and polite and respond in a timely manner" can become "Be concise and professional."

### JSON payloads

Compact JSON uses significantly fewer tokens than pretty-printed:

```json
{"key":"value","count":1}
```

vs:

```json
{
  "key": "value",
  "count": 1
}
```

### RAG context

Limit retrieved document chunks to the minimum needed. Truncate each chunk to the first N characters. Use overlap only when semantic continuity is important.

### Conversation history

Summarize older messages rather than passing the full history. Use a rolling window that keeps only the last N turns.

### Whitespace

Remove trailing spaces, multiple blank lines, and repeated whitespace. This has a measurable impact on large prompts.

## Cost formula

```
input_cost  = (input_tokens  / 1_000_000) * input_price_per_million
output_cost = (output_tokens / 1_000_000) * output_price_per_million
total_cost  = input_cost + output_cost
```

Monthly projection:

```
monthly_cost = total_cost * requests_per_day * 30
```

## Token counting in code (Node.js)

```typescript
import { get_encoding } from 'js-tiktoken';

const enc = get_encoding('cl100k_base');
const tokens = enc.encode('Hello, world!');
console.log(tokens.length);  // 4
enc.free();
```

## Token counting in code (Python)

```python
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("Hello, world!")
print(len(tokens))  # 4
```