AI & LLMs Runs locally

LLM Tokenizer Visualizer

See the real BPE tokens, token IDs, and character ratio for pasted text.

ELI5

It shows how an AI breaks your text into the small pieces it reads.

A little more detail

What this tool does

AI models do not read whole words the way people do. They split text into tokens, then charge and enforce limits based on those tokens. This tool lets you inspect that split before you send text to a model.

Use it to

See why code, JSON, or unusual words use more tokens than expected
Estimate how much room a prompt will take in a context window
Compare tokenization across supported encodings

Interactive workspace Results update as you type

Your text never leaves the browser — no upload, no server. Prompt text is not stored in the URL.

Encoding

Model caveat: Used by: . Token counts are exact for this encoding, not a rough char/4 estimate.

Text

Loading encoding and tokenizing…

Stats

tokens chars unique IDs avg chars/token roundtrip ✓ roundtrip mismatch

Visual

Enter text above to see token spans.

Token IDs

LLM APIs bill and truncate by tokens, not characters. BPE (byte-pair encoding) merges frequent byte sequences into token IDs. The same word can split differently across encodings or even positions in a prompt.

This tool uses the gpt-tokenizer library with real rank tables — not a chars÷4 guess. Pick the encoding that matches your model family; when unsure, o200k_base covers GPT-4o and newer chat models.