Distillr

Trim what is irrelevant, encode what is left in a token-efficient format, and know exactly how much you saved and whether it was safe.

Every team sending structured or semi-structured data to an LLM pays a syntax tax: repeated JSON keys, irrelevant rows, stale chat turns, verbose formatting. Point tools exist for pieces of this. Distillr combines them into one pipeline with per-stage accounting and an audit trail of everything it removed.

payload ──▶ Stage 1 retrieve/trim ──▶ Stage 2 semantic (opt.) ──▶ Stage 3 encode (auto) ──▶ Stage 4 audit ──▶ prompt
             drop empty columns         LLMLingua-2                TOON / JSON / CSV,         manifest of every
             BM25 rank by query                                    whichever is cheapest      removal + risk
             keep recent chat turns                                on your tokenizer
                                        └──────────── token ledger: before/after per stage, per run ────────────┘

Quickstart

pip install distillr           # or: git clone ... && pip install -e '.[dev]'
distillr analyze orders.json --query "orders for Ada Lovelace shipped to Berlin" --top-k 12 --show
distillr ledger                # what you have saved so far

import distillr

result = distillr.compress(rows, query="orders shipped to Berlin", top_k=20)  # picks the cheapest lossless encoding
prompt = f"Answer from this data:\n{result.text}"
print(result.tokens_before, "->", result.tokens_after, f"({result.savings_pct:.0f}% saved)")

answer = llm(prompt)
for flag in result.check_answer(answer):  # did the model lean on something we cut?
    print(flag.risk, flag.path, flag.matched)

Stages are composable:

from distillr import Pipeline, RetrieveStage, EncodeStage, AuditStage

pipe = Pipeline([RetrieveStage(keep_fields=["id", "name", "*_at"], top_k=50), EncodeStage("csv"), AuditStage()], model="gpt-4o")

What each stage does

Stage	Purpose	Lossless?	Records
retrieve	Drop columns that are empty in every row (per-cell removal would break the tabular layout), dedupe, field allow/deny lists, truncate long strings, BM25-rank items against your query (or an embedding scorer you pass in), keep system + recent chat turns	No, by design	one `Removal` per field, item, or truncation with a preview
semantic	LLMLingua-2 token pruning at a target rate (`pip install 'distillr[semantic]'`)	No	pruned tokens
encode	Re-serialize as TOON, compact JSON, or CSV. `auto` (default) encodes with each and keeps the one with the fewest tokens on the active tokenizer; `--flatten` turns nested objects into dotted columns so more arrays qualify for TOON's tabular form	Yes, `distillr decode` proves it	format chosen, candidates
audit	Summarize the manifest by risk; `check_answer()` flags answers that reference removed content	n/a	risk counts

Token counts use real tokenizers (tiktoken o200k_base by default, --model claude-sonnet-5 picks the closest and flags approximations). Counts are never estimated silently.

Benchmarks (Phase 0 gate: passed)

Five realistic payload types, each with planted facts the downstream question needs. Savings are measured with tiktoken o200k_base against the payload as a team would paste it (pretty JSON / JSONL). Needle recall is the share of those facts that survive verbatim; it must stay at 100%.

Case	Tokens before	After	Retrieve	Encode (format)	Saved	Recall
200 e-commerce orders, query "orders for Ada Lovelace shipped to Berlin", top 12	47,352	1,795	94.0%	36.5% (json)	96.2%	100%
60-turn support chat, keep system + last 6 + 12 relevant	5,301	337	90.4%	33.5% (toon)	93.6%	100%
40 RAG chunks, query "sev-1 SLA status page", top 5	4,622	409	88.0%	26.2% (json)	91.2%	100%
Nested org/members API response, drop url/id noise, flatten	9,334	2,018	55.5%	51.5% (toon)	78.4%	100%
400 JSONL log lines, query "payment-service timeout", top 8	36,379	388	97.9%	49.0% (csv)	98.9%	100%
Total	102,988	4,947			95.2%	100%

Two honest notes. Most of the saving comes from Stage 1: sending the right rows matters more than how you spell them. And TOON only beats compact JSON when rows are uniform and flat; on nested or sparse records JSON wins, which is why auto measures instead of assuming. Full table with the format layer isolated: benchmarks/RESULTS.md, regenerated by distillr bench.

Positioning

Tool	What it does	What Distillr adds
LLMLingua	Semantic token pruning with a small LM	A retrieval stage before it, a format stage after it, a ledger and audit trail around it. LLMLingua-2 is Distillr's Stage 2, not a competitor
TOON / TRON	Compact serialization	Deciding what to send before encoding; measuring what the encoding actually saved on your tokenizer. TOON is Distillr's default Stage 3
leanctx, llmslim	Single-technique SDKs	Composable stages, CLI, ledger, audit survivability
LangChain, LlamaIndex	Retrieval frameworks	Distillr plugs in after retrieval and accounts for tokens; it does not replace your vector store

The token ledger

Every run is recorded in SQLite (~/.distillr/ledger.db, override with DISTILLR_LEDGER): tokens before and after per stage, payload kind, encoding, tokenizer, the removal manifest, and any audit flags. distillr ledger summarizes; distillr ledger --export runs.json dumps it. The hosted dashboard (Phase 2) reads the same schema from Postgres.

Status and roadmap

Phase 0 (this release): Stage 1 + Stage 3 + ledger + CLI + benchmarks. Stage 2 and Stage 4 interfaces are in place; LLMLingua-2 is optional.

Phase 1: LLMLingua-2 wired into the default pipeline, Python SDK polish, self-hosted OpenAI-compatible proxy (Docker), public release
Phase 2: hosted proxy, dashboard on the ledger, usage-based billing
Phase 3: multi-provider routing, semantic caching, TS/JS SDK, enterprise features

Self-hosted and OSS stay free. Full plan with issues open to contributors: ROADMAP.md. Spec and design notes: docs/spec.md, docs/design.md. Site: https://dwarka-prasad.github.io/distillr/

Development

python -m venv .venv && . .venv/bin/activate
pip install -e '.[dev]'
make test lint bench

Apache-2.0.