v0.1.0 · Apache-2.0 · Python · – stars

Cut LLM token spend without dropping what the model needed.

Distillr sits between your app and your LLM provider. It trims the rows and fields that don't matter for this request, re-encodes what's left in the cheapest lossless format for your tokenizer, and keeps a ledger of every token saved and every field removed.

Get started Benchmarks GitHub

$ distillr analyze orders.json --query "orders for Ada Lovelace shipped to Berlin" --top-k 12 orders.json kind=tabular tokenizer=o200k_base Stage Tokens in Tokens out Saved % Notes retrieve 47,352 2,828 44,524 94.0% kept 12/200 items · 190 removals encode 2,828 1,795 1,033 36.5% json · auto: toon=1,836 json=1,795 · lossless audit 1,795 1,795 0 0.0% risk high=3 med=185 low=2 total 47,352 1,795 45,557 96.2% 48 ms $ distillr ledger 6 runs · 102,988 tokens in · 4,947 out · 98,041 saved (95.2%)

The problem: you pay a syntax tax on every request

Teams paste JSON API responses, whole chat histories and top-40 RAG chunks into prompts. Most of it is irrelevant to the question, and the rest is spelled in the most expensive way possible: repeated keys, nulls, indentation. Point tools exist for pieces of this (LLMLingua prunes tokens, TOON encodes compactly, RAG frameworks retrieve) but nothing combines them, measures each stage honestly, or tells you when a cut was unsafe.

200 orders + query

96.2%

60-turn support chat

93.6%

40 RAG chunks + query

91.2%

Nested API response

78.4%

400 JSONL log lines

98.9%

Bar = tokens remaining after Distillr, relative to the payload as pasted. tiktoken o200k_base. Needle recall 100% in every case.

How it solves it: four stages, one ledger

Each stage is a pure function from data to (data, report). Every removal carries a path, a reason and a preview, so the audit stage can later tell whether the model leaned on something you cut.

Retrieve / trimDrop columns empty in every row, dedupe, allow/deny fields by glob, truncate long strings, rank items against your query with BM25 (or your embeddings), pin system and recent chat turns.

Semantic (optional)LLMLingua-2 token pruning at a target rate, structure tokens protected. Install with pip install 'distillr[semantic]'.

EncodeTOON, compact JSON or CSV. auto encodes with each and keeps the fewest tokens on your tokenizer. Lossless: distillr decode proves it.

AuditSummarizes the manifest by risk. result.check_answer(answer) flags answers that reference removed content but not kept content.

How a request flows

One payload, four stages, one ledger. Numbers are the real benchmark run on 200 orders with the query "orders for Ada Lovelace shipped to Berlin". Bars show tokens remaining; every stage also writes what it did to the ledger.

tokens remaining, bar on a square-root scalelossless stageoptional stage

Two lines to integrate

Use the CLI to try it on a file, the SDK in your app, and the ledger to see what you saved over time.

import distillr

result = distillr.compress(rows, query="orders shipped to Berlin", top_k=20)
prompt = f"Answer from this data:\n{result.text}"
print(result.tokens_before, "->", result.tokens_after, f"({result.savings_pct:.0f}% saved)")

pip install distillr
distillr analyze payload.json --query "refund for order ORD-55213" --top-k 12 --show
distillr analyze members.json --drop "*_url,node_id" --flatten
distillr ledger --days 7

from distillr import Pipeline, RetrieveStage, EncodeStage, AuditStage

pipe = Pipeline(
    [RetrieveStage(keep_fields=["id", "name", "*_at"], top_k=50, scorer=my_embeddings),
     EncodeStage(format="auto", flatten=True),
     AuditStage()],
    model="claude-sonnet-5",
)
result = pipe.run(payload, query=user_question)

answer = llm(prompt)
for flag in result.check_answer(answer):
    print(flag.risk, flag.path, "answer mentions", flag.matched, "which was removed:", flag.reason)

# high  rows[88].shipping.tracking  answer mentions trk777000001  which was removed: low relevance (score 0.42)

Where it fits

Distillr does not replace the pieces. It composes them and adds what none of them have.

Capability	LLMLingua	TOON	leanctx / llmslim	RAG frameworks	Distillr
Decide what to send (retrieval trim)	no	no	partial	yes	yes
Semantic token pruning	yes	no	yes	no	via LLMLingua-2
Token-efficient lossless encoding	no	yes	no	no	TOON / JSON / CSV, measured
Per-stage token accounting	no	no	no	no	ledger, SQLite or Postgres
Audit trail of removed content	no	no	no	no	manifest + check_answer
CLI with zero integration	no	yes	no	no	distillr analyze

Roadmap

Phase 0 (this release) validated the claim. What comes next follows the spec's order: prove the OSS engine, then the hosted wedge.

Phase 1 · OSS release

LLMLingua-2 in the default pipeline, Python SDK polish, self-hosted OpenAI-compatible proxy in Docker, PyPI release.

Phase 2 · Hosted wedge

Hosted proxy with zero infra, dashboard on the ledger (savings over time, per endpoint, audit-risk trend), usage-based billing.

Phase 3 · Expand

Multi-provider routing to the cheapest capable model, semantic caching, TypeScript SDK, SSO and on-prem for enterprise.