token-counter

# token-counter

Count tokens in text using the cl100k_base tokenizer (used by GPT-4, GPT-4o, GPT-3.5-turbo). Runs entirely in the browser via tiktoken WASM - no server required.

## When to use

- Checking if a prompt fits within a model's context window
- Comparing token counts of two prompt versions
- Estimating API costs before making requests
- Visualizing exactly how a tokenizer splits text
- Saving and comparing snippets across sessions

## Running locally

```bash
git clone <repo> token-counter
cd token-counter
pnpm install
pnpm dev
# Open http://localhost:5173
```

Production build:

```bash
pnpm build
pnpm preview
```

No environment variables required. No server. All tokenization runs in the browser.

## Pages

| Route | Description |
|---|---|
| / (Counter) | Main text input and token visualization |
| /compare | Side-by-side comparison of two texts |
| /snippets | Saved snippets list |
| /cost | Cost estimator across all models |
| /guide | Tokenizer reference guide |

## Supported models

| Model | Tokenizer | Input per 1M | Output per 1M |
|---|---|---|---|
| gpt-4o | cl100k_base (exact) | $5.00 | $15.00 |
| gpt-4 | cl100k_base (exact) | $30.00 | $60.00 |
| gpt-4-turbo | cl100k_base (exact) | $10.00 | $30.00 |
| gpt-3.5-turbo | cl100k_base (exact) | $0.50 | $1.50 |
| claude-3-5-sonnet | ~4 chars/token | $3.00 | $15.00 |
| claude-3-opus | ~4 chars/token | $15.00 | $75.00 |
| gemini-1.5-pro | ~4 chars/token | $1.25 | $5.00 |
| llama-3-70b | ~4 chars/token | $0.59 | $0.79 |

Models marked "~4 chars/token" use a character-based approximation since they do not use cl100k_base. The UI labels these as "(approximate)".

## Token visualization

Each token in the visualization panel is rendered as a colored `<span>` with a cycled 20-color palette. When "Show token IDs" is enabled, the integer vocabulary index is displayed below each span.

Example:
- "tokenizer" splits into: `token` (ID: 4037) + `izer` (ID: 7609) = 2 tokens
- "Hello" = 1 token (ID: 15339)
- " world" (with leading space) = 1 token (ID: 1917)

## Keyboard shortcuts

| Shortcut | Action |
|---|---|
| Ctrl+Enter | Save current text as snippet |
| Ctrl+K | Clear the text input |

## localStorage

| Key | Description |
|---|---|
| token-counter-snippets | JSON array of saved Snippet objects |
| token-counter-settings | Settings object (default model, output ratio, etc.) |

No data is sent to any server. All storage is local to the browser.

## Cost estimation

The Cost page allows configuring:
- Input token count (manual entry or pulled from Counter)
- Output ratio (output = input x ratio, default 0.5x)
- Requests per day (for monthly projection)

All 8 models are compared in a sortable table. The cheapest model for the given token count is highlighted in green.

## Tips for reducing token usage

- Remove extra whitespace and blank lines
- Use compact JSON (no indentation)
- Keep system prompts under 200 tokens
- Use common, short words instead of rare or long ones
- Truncate retrieved context chunks in RAG pipelines

See the Guide page (/guide) for a full explanation of how cl100k_base works and common token counts by content type.