Token-Native Storage

Looking for TL;DR? Check key takeaways

Every vector search engine stores two things for each point: the vector and the payload. The AI community has gone all in to compress the vectors with quantization and MRL. However, the text payload gets raw UTF-8, or LZ4 at best, even though the systems that actually consume it (embedders, rerankers, LLMs) operate on tokens, not bytes. I realised that using OpenAI r50k token IDs over raw text already achieves 2.33× compression, beating every byte-level codec, and an entropy-based compressor on top of the token IDs reaches 3.37× (beating even a corpus-trained zstd). Effectively, tokenization gives you free compression!

In other words, if agents are going to be the biggest users of your system, then your system should speak their language (tokens), so you're not paying a translation cost on every read. This post argues for token-native storage (read/write text as BPE token IDs, not UTF-8 bytes) anywhere models are the primary readers and writers: search engines, LLM agent traces, and maybe even local files. I'll show my point with experiments, talk about limitations, and how the AI/DB community (and the AI labs) can help get rid of them.

How Token-Native Storage Works

The top path is what databases do today: store UTF-8 bytes, compress with a generic codec. The bottom path is token-native storage: tokenize first, then compress (here entropy-code) the token IDs.

Step 1: BPE Tokenization

BPE (Byte Pair Encoding) starts with raw bytes and repeatedly merges the most frequent adjacent pair into a new token, building up a vocabulary of common subwords. OpenAI has three popular BPE tokenizers:

1. r50k: used by GPT-2/GPT-3, 50,257 tokens and fits in uint16 (= ~65k)
2. cl100k: used by GPT-3.5/GPT-4, 100,277 tokens and fits in 17 bits (131k) but only 24 bits (3 bytes) is practical due to overhead
3. o200k: used by GPT-4o/GPT-4o mini/o1/o3, 200,019 tokens, same 24-bit (3-byte) practical packing as cl100k

BPE merges frequent byte sequences into single tokens. "storage" is one token. "Token-native" is three (Token + - + native). On average one BPE token covers 3/4 of a word. An average English word is ~5 characters plus a space, so:

Using UTF-8 text: 6 bytes/word (1 char = 1 byte) × 3/4 word/token = 4.5 bytes/token (raw text)
Using Tokens: 2 bytes/token (r50k uint16 token IDs)
ratio: ~2.25x

This napkin math is what gave me the motivation to run the experiment, and my experiments below prove it in practice.

Another fun fact is that a every word in a sentence comes with a " ", so using uint16 token performs better for any word longer than 1 character. Example: " cat" is 4 raw bytes (space + 3 letters) but just 1 token, 2 bytes (uint16), an easy win :D

Step 2 (Optional): ANS Entropy Coding

Step 1 alone already gets you the ~2.25× above, raw uint16 packing with no compression algorithm running at all. This step is for squeezing further: gzip and LZ4 see opaque bytes and can't know that [0x01, 0x7F] is token 383. An entropy coder working directly on token IDs can give frequent tokens short codes: "the" is 40x more common than "embeddings", so it gets a much shorter code. I used ANS (Asymmetric Numeral Systems, the entropy coder inside zstd) via the constriction library.

There's one important thing to understand here: the frequency table is trained once on a large corpus and shared, not built per document. A per-document table compresses better on paper (it achieved 5x compression), but the decoder needs the table to reconstruct the document, so you'd have to store it alongside each payload. That's ~900 bytes for a typical 512-token chunk, which defeats the purpose.

import constriction, numpy as np, tiktoken

enc   = tiktoken.get_encoding("r50k_base")
VOCAB = 50_257

# ── one-time setup: train on corpus ──────────────────────────────────────────
corpus_ids = enc.encode(corpus_text)
counts = np.bincount(corpus_ids, minlength=VOCAB).astype(np.float64) + 1  # Laplace
probs  = counts / counts.sum()
model  = constriction.stream.model.Categorical(probs, perfect=False)

# ── per-document encode (store this) ─────────────────────────────────────────
def compress(text):
    ids   = enc.encode(text)
    coder = constriction.stream.stack.AnsCoder()
    coder.encode_reverse(np.array(ids, dtype=np.int32), model)
    return len(ids).to_bytes(4, 'big') + coder.get_compressed().tobytes()

# ── per-document decode ───────────────────────────────────────────────────────
def decompress(data):
    n   = int.from_bytes(data[:4], 'big')
    buf = np.frombuffer(data[4:], dtype=np.uint32).copy()
    ids = constriction.stream.stack.AnsCoder(buf).decode(model, n).tolist()
    return enc.decode(ids)

Benchmarks

Storage efficiency

Measured across 1,773 WikiText-103 test articles (≥30 words). The ANS frequency table was trained on the WikiText-103 train split, a shared static model with no per-document overhead. All methods below are lossless (0% character error rate on full round trip).

Observations:

1. r50k raw uint16 already beats every byte codec compression algorithm!
2. r50k + static ANS beats everything by a wide margin.
3. cl100k and o200k just need the right container: switching from naive uint32 (4-byte) packing (left off the chart above, see the glossary for why) to 3-byte packing, with no entropy coder at all, already gets them to 1.49x and 1.52x. ANS recovers further still, to 3.30x (cl100k) and 3.32x (o200k).
4. The fairest competitor to r50k+ANS is zstd --train, since it also learns a vocab from the corpus. It reaches 2.61x, still 30% behind! (and its vocab isn't standardized like the tokenizers)

But WikiText is one curated corpus, so I reran everything on real prose/code/Hindi corpora at realistic RAG chunk sizes in a follow-up post. Prose and code hold up fine (~3.3-3.4x, ~3.1-3.2x). Hindi is the fun one: r50k and cl100k raw-pack it to 0.84-0.87x, worse than doing nothing, because neither ever learned real Devanagari merges. o200k, which did learn them, gets 5.5-6.1x on the exact same text. Same method, opposite outcome, all down to whether your tokenizer actually covers your script.

Latency

Measured across 200 WikiText-103 articles (50–500 words), static corpus ANS model:

Observations:

1. LZ4 compresses in 15µs and decompresses in 1µs (it's designed entirely for speed!)

2. Tokenization is expensive but it's anyways a mandatory step to pass text to the model so it's free lunch and more importantly, re-usable. Detokenization is fast so it's easy to also show text back to humans.

3. ANS decode looks slow next to LZ4's raw decompress, but that's not a fair fight: an agent needs token IDs either way, so a byte-codec payload still owes it a tokenize step after decompressing. Charge that fairly and ANS decode comes out 2-8x faster (measured across prose/code/Hindi). It's still slower than the frequency-sorted alternative below, just not "slow" on its own.

4. zstd dict's encoder is surprisingly slow at 128µs, though it decompresses in 2µs.

Note that for RAG and agentic search, many of today's embedding models still use WordPiece or SentencePiece instead of LLM BPE tokenizers, so you can't skip tokenization for most embedding/re-ranking models. However, that's just BERT-family legacy, not a fundamental problem. OpenAI's text-embedding-3 family already uses cl100k directly. Jina's embeddings-v4 moved to a real BPE tokenizer too, inherited from its Qwen2.5-VL backbone. Either way, storing token IDs in LLM format (r50k/cl100k) still avoids re-tokenizing retrieved documents at inference time. It's still a win :)

Note that every copy of your data: snapshots, backups, write-ahead logs, replication, and egress enjoy the same 3.37× reduction, multiplying benefits beyond storage: less data moves over the network, more fits in RAM, and reads and indexing get faster.

A Faster Alternative to ANS: Frequency-Sorted Token IDs

Worth separating two things at this point. Storing token IDs instead of bytes, the core argument above, is settled. Which entropy coder you put on top of it is a separate, still-open question, this section is my current answer, not the final one.

ANS gets close to the entropy limit, but it's just one option, and it turned out to be slow to decode. But then I realised that BPE assigns token IDs in merge-discovery order, not by real-world frequency, so fixed-width packing wastes that gap, 2 bytes is 2 bytes whether the ID is 5 or 50,000. The fix is simple: reassign IDs by frequency rank, then use a variable-length integer code.

No fixed width fixes this, wider just wastes more (see the glossary above for why cl100k/o200k need 3 bytes at all). Reassigning IDs by frequency rank and packing with streamvbyte (a real SIMD-vectorized library from the FastPFOR family) gets most of the way to ANS's ratio with no entropy coder at all, at a ~15-20x faster decode, a genuine, deployable middle ground.

This holds up beyond WikiText too. I re-measured it on prose/code/Hindi and frequency-remap decodes 1.5-5.7x faster than ANS every time, for about 15-26% less ratio. So it comes down to what you care about: decode latency (agents on the hot path) points you to frequency-remap, ratio (cold storage, egress) points you to ANS. Or just run both, ANS at rest and frequency-remap for a hot cache, since under the hood they're both just token IDs.

What AI Labs Could Fix for Free

To OpenAI, Anthropic, Alibaba, DeepSeek, Google, and anyone else training the next BPE vocabulary: assign token IDs by frequency rank instead of merge-discovery order. It costs nothing extra, it's the same vocabulary either way, just a different choice of which integer means which token, and it hands every downstream user some extra token compression by default.

You don't have to wait around for that, though. The remap is just a lookup table you can build yourself. Apply it in your own storage layer today, no changes to the tokenizer or the model required, and honestly you'll do better than a vendor's version anyway: yours is fit to your corpus, not some generic training mix.

What This Looks Like in Practice

Here's the actual API, not just benchmark numbers. One-time setup: register a tokenizer on the field, at the collection level.

PUT /collections/my_collection/index
{ "schema": { "text": { "type": "token", "tokenizer": "r50k" } } }

After that, insert barely changes: text still takes a plain string and the engine tokenizes it, or you hand it token IDs directly as an array and skip tokenization entirely.

PUT /collections/my_collection/points
{
  "points": [{
    "id": 123,
    "vector": [0.12, -0.34, ...],
    "payload": { "text": [1858, 6427, 20272, 318, 257, 6997, ...] } // or a plain string
  }]
}

A string or an array is easy to tell apart, so the database picks the right path, tokenize-then-compress or compress directly. ANS happens on write and reverses on read, invisibly. The tokenizer is a collection-level choice, not per-document: mixing tokenizers means mixing vocabularies, and the static table only makes sense for one.

Reading needs the same symmetry, so with_payload takes a format per field instead of just a boolean:

POST /collections/my_collection/points/search
{ "vector": [...], "limit": 10, "with_payload": { "text": "tokens" } }
// payload comes back as {"text": [1858, 6427, 20272, 318, 257, 6997, ...]}

Default stays "text", so existing clients see no change. An LLM-facing pipeline asks for "tokens" and skips detokenization; anything human-facing asks for "text". This only closes the loop end to end if you own the inference stack, since hosted LLM APIs take text prompts, not token arrays (yet).

An LLM produces token IDs before the inference engine detokenizes them, so storing generated output (chat history, agent thinking traces) this way has zero encode cost. It splits elegantly by workload: writing-heavy (agent traces, synthetic data) gets free token IDs, reading-heavy (RAG, agent memory) skips retokenizing on future reads. Human reads stay cheap either way, detokenizing is ~4-12µs.

Limitations

• No shared vocabulary across models yet. Embedding models use WordPiece or SentencePiece, LLMs use BPE variants (r50k, cl100k, o200k), and different LLM families don't share one either. A token-native payload only pays off end-to-end for a consumer using the same tokenizer it was stored with. That's exactly the case for a common token vocabulary standard, the way ASCII and UTF-8 standardized bytes, rather than treating token-native storage as a single-vendor optimization.
• Hosted LLM APIs don't accept or return token IDs today. Even when your storage tokenizer matches the model's exactly, the read path only closes the loop end to end if you own the inference stack: hosted chat APIs take text prompts and return text, not raw token arrays, so a detokenize/retokenize step still happens somewhere. See this follow-up proposing a token-ID-native LLM API for what closing that gap would actually take.
• Raw packing efficiency depends on the tokenizer and the script. For a script a tokenizer has no merges for, packing can lose outright: r50k never learned to merge Devanagari bytes, so the Hindi word भारत (12 raw UTF-8 bytes) falls back to 7 single-byte tokens, 14 bytes as uint16, worse than doing nothing. Storing tokens is still worth it if an LLM is going to consume them either way, since it skips re-tokenizing at read time regardless of the ratio.
• ANS is one entropy coder, not the point of this post, and its cost is asymmetric. It's what got the benchmark from 2.33x to 3.37x, but decode is slower than LZ4 (see latency) and recurs on every read, unlike the one-time encode, exactly what motivates the frequency-sorted-ID alternative above. Whichever coder or remap you use, the frequency data has to match your actual corpus: a WikiText-trained ranking applied to JSON gets 1.08x, a JSON-trained one gets 1.99x. The core argument, store tokens instead of bytes, holds regardless of which coder, if any, you layer on top.

Key Takeaways

• Switching from UTF-8 bytes to token IDs brings you free compression. OpenAI r50k tokenizer hits 2.33x, already beating every byte codec. ANS entropy coding on top gets you to 3.37x.
• The compression gains are generalizable and across all tokenizers from all vendors. But another massive bonus is never having to tokenize again as models read your data thousands or millions of times in the future.
• Although token-native storage is already usable and beneficial, the ecosystem needs to evolve to make it even better: models don't share a vocabulary yet, and hosted LLM APIs still only take text in and text out.

Acknowledgements

The benchmarks use tiktoken, zstandard, lz4, and constriction. Thanks to the concept of napkin math for making me realise this gap. Apparently, the "language modeling is compression" is somewhat explored in DeepMind's 2023 paper if you want further read.

Citation

If you find this useful, please cite:

@misc{kumar2026tokenstorage,
  author       = {Kumar Shivendu},
  title        = {Token-Native Storage: Read and Write in the Language Models Already Speak},
  year         = {2026},
  url          = {https://www.kshivendu.dev/blog/token-storage},
  note         = {Blog post}
}