Context Compression: How to Count Tokens, What to Compress and What to Keep, and How to Verify Nothing Was Lost

Learn to count before learning to save: character estimates, tokenizers, real usage

To save you must first count. There are three ways to count tokens, each with its use, and mixing them causes errors.

One: exact counting, running the model's own tokenizer. The only truly accurate one. But this course connects to any OpenAI-compatible gateway, behind which may be any model from anyone — the token table is not unique. Installing a tokenizer package gives only an illusion of exactness: precise arithmetic over someone else's tokenization.

Two: real usage, reading the gateway's returned usage. The prompt_tokens in that final usage event is what this request actually cost. It is accurate, and it is after the fact: judging "will this exceed" needs a number before sending.

Three: local estimation. Simple in shape, two terms added:

tokens ~= a x CJK character count + b x other character count

Day eight's accounting line used exactly this, with a as 1 and b as one quarter — roughly one token per CJK character, roughly one per four Latin characters. Enough to print "about 86 tokens," and no further.

Their division in one sentence: estimation decides (before a request, should we compact), real usage reconciles (after a request, how good was the estimate), exact counting settles (when you actually bill by token, which is day twenty).

Calibrate the estimator with returned usage

Since every request returns real usage, the coefficients should not be guessed but regressed.

The method collects samples in passing: before sending, join the message array into text and record it; when usage returns, pair "CJK character count, other character count" with "real token count" as one sample. With three samples, solve a least-squares fit — two unknowns, two normal equations.

Collect in passing, never send extra requests to measure. Dedicated probe requests cost money and measure a different kind of text; what you want to estimate well is the kind your program actually sends.

fit(minSamples = 3): FitResult {
  const before = meanAbsPercentError(this.samples, active)
  if (this.samples.length < minSamples) return notEnough(before, this.samples.length)
 
  // The six sums of the two normal equations: sum cjk^2, cjk*rest, rest^2, cjk*t, rest*t
  let sxx = 0, sxy = 0, syy = 0, sxt = 0, syt = 0
  for (const s of this.samples) {
    sxx += s.cjk * s.cjk
    sxy += s.cjk * s.rest
    syy += s.rest * s.rest
    sxt += s.cjk * s.tokens
    syt += s.rest * s.tokens
  }
 
  const det = sxx * syy - sxy * sxy
  // A relative criterion, not an absolute one: with many samples, sxx*syy is itself huge
  if (Math.abs(det) < 1e-6 * Math.max(1, sxx * syy)) return flatFallback(this.samples, before)
 
  const coeff = {
    cjk: clamp((sxt * syy - syt * sxy) / det),
    rest: clamp((syt * sxx - sxt * sxy) / det),
  }
  return { coeff, samples: this.samples.length, errorBefore: before,
    errorAfter: meanAbsPercentError(this.samples, coeff), degenerate: false, note: 'calibrated' }
}

def fit(self, min_samples: int = 3) -> FitResult:
    before = mean_abs_percent_error(self.samples, ACTIVE)
    if len(self.samples) < min_samples:
        return not_enough(before, len(self.samples))
 
    # The normal equations' coefficient matrix and right-hand side, gathered in one pass
    sxx = sum(s.cjk * s.cjk for s in self.samples)
    sxy = sum(s.cjk * s.rest for s in self.samples)
    syy = sum(s.rest * s.rest for s in self.samples)
    sxt = sum(s.cjk * s.tokens for s in self.samples)
    syt = sum(s.rest * s.tokens for s in self.samples)
 
    det = sxx * syy - sxy * sxy
    if abs(det) < 1e-6 * max(1, sxx * syy):   # the CJK-to-other ratio barely varies
        return flat_fallback(self.samples, before)
 
    coeff = Coefficients(
        cjk=clamp((sxt * syy - syt * sxy) / det),
        rest=clamp((syt * sxx - sxt * sxy) / det),
    )
    return FitResult(coeff, len(self.samples), before,
                     mean_abs_percent_error(self.samples, coeff), False, "calibrated")

The only subtlety is that det check: a determinant near zero means the samples' CJK-to-other ratio barely varies (most commonly all-Latin text), and the two coefficients are then mathematically inseparable. Forcing a solution yields an absurd pair (often one positive and one negative), worse than no calibration, because it looks calibrated. So that branch must admit it cannot solve and fall back to one uniform per-character coefficient.

The self-test calibrates on five texts with different ratios and prints these two lines:

v calibrated on 5 samples: coefficients CJK 0.50 / other 0.51, error 55.6% -> 2.6%
v an all-ASCII sample judged unsolvable: the CJK-to-other ratio is too uniform to separate the two coefficients; falling back to one per-character coefficient

Both lines are reproducible offline, but offline "real usage" comes from the script (derived from character counts), so the error drops flatteringly; against a real gateway not one line of code changes, the regression yields different coefficients, and the error will not be that low.

One last discipline: the calibration result is one process-level table shared by all accounting, or the same text is reported as two different numbers in two places.

When to trigger: by proportion, not by turn count

The obvious criterion is turns: compact after twenty. It is wrong — there is no stable relationship between turns and occupancy. One grep matching two thousand lines eats half the window in a turn; ten turns of "change a line, run a test" may not reach ten percent. Turn-based triggering produces two behaviors only: compacting too early on short conversations and too late on long results (remembering to compact after already exceeding).

The correct criterion is proportion: compact when the estimate exceeds seventy percent of the total budget.

Which requires a "total budget" table. Each path reported its own ledger over the past days; today is where they are totaled, and the only place in the course that discusses dividing the total. The idea in one sentence: give the permanent paths fixed allowances and leave the rest to the ones that grow.

Fixed allowances are not about saving money but about naming who overran: with one total you know only that it is full; with per-path allowances you know which path to compact. The lab's /budget prints this table, listing even the empty paths with their allowances.

The trigger sits at seventy rather than ninety percent for a practical reason: compaction itself takes room — the summarization request resends the conversation being compacted, and the summary occupies space too. Compacting at ninety percent risks the compaction request itself exceeding the limit.

The self-test pins this as an assertion: the same fourteen messages, the same turn count, differing only in tool result length —

v same 14 messages, same turns: the long one triggers at 1368 tokens, the short one does not at 176

What to compress: consecutive tool round trips

What can be compressed is not homogeneous: within the same history, a thousand tokens of tool results and a thousand tokens of user requirements differ by an order of magnitude in compression value.

Consecutive tool round trips are most worth compressing, for three increasingly practical reasons:

One, they occupy the most. Reading back hundreds of numbered lines, a test output of over a thousand characters — tool results are almost always the largest block.

Two, their information has already been digested. The model read the file and said "divide has no divide-by-zero guard," and that conclusion is in the assistant message right after; keeping the original only pays twice for one conclusion.

Three, they can be re-fetched. The file is still on disk and the command can run again. The worst case of compressing it is reading it again when needed; compressing a user requirement loses that sentence forever — they will not repeat it, they assume you remember.

So priority follows "can it be re-fetched," not recency. That transfers directly: compress the re-fetchable (files, search results, web bodies) first and the unrecoverable (the user's words, one-time responses, random ids) last.

What to keep: an explicit keep list

What to keep must be a whitelist: state which categories survive verbatim, and the rest is compressible. Enumerating "what may be compressed" will eventually miss a newly introduced message type, and the consequence is silent information loss — harder to spot than day eight's red line: a lost injection leaves an accounting line, a lost compaction leaves nothing.

The keep list has four entries:

1. System instructions. They are this program's persona and rules; compressing them changes the program.
2. The todo checklist. It is the sole record of "how far I have got," and summarizing it makes the model redo finished work. The checklist is day ten's mechanism, but the keep list must reserve its place first: the lab's criterion is a prefix match, so any checklist starting with those characters is automatically protected.
3. Unfinished tool calls. Fewer tool results than calls means the previous turn was interrupted. A summary line saying "called run_command" cannot replace the missing result — what the model needs is to fill it in.
4. The most recent few groups. Everything the model is currently doing is in them. A summary is a conclusion; right now it needs the original. Compressing them typically makes it repeat work just done.

And one rule harder than the list: an assistant message with tool calls and all its tool results are inseparable — the same origin as day seven's fork rule; splitting them yields a message array where a tool was requested with no result, making the next request invalid outright. So the first step is not selecting messages but grouping:

export function groupMessages(messages: Message[], options = DEFAULT_COMPACT_OPTIONS): Group[] {
  const groups: Group[] = []
  for (let i = 0; i < messages.length; i += 1) {
    const message = messages[i] as Message
    if (message.role === 'assistant' && message.toolCalls?.length) {
      // Swallow the consecutive tool messages that follow into this group
      let end = i + 1
      while (end < messages.length && messages[end]?.role === 'tool') end += 1
      const results = end - (i + 1)
      // Fewer results than calls means the previous turn was interrupted: keep this group verbatim
      groups.push({ start: i, end, keep: results < message.toolCalls.length ? 'unfinished' : 'compressible' })
      i = end - 1
      continue
    }
    groups.push({ start: i, end: i + 1,
      keep: message.role === 'system' ? 'system' : options.pinned(message) ? 'pinned' : 'compressible' })
  }
  return groups
}

def group_messages(messages: list[Message], options: CompactOptions = DEFAULT) -> list[Group]:
    groups: list[Group] = []
    i = 0
    while i < len(messages):
        message = messages[i]
        if message.role == "assistant" and message.tool_calls:
            end = i + 1
            while end < len(messages) and messages[end].role == "tool":
                end += 1
            results = end - (i + 1)
            keep = "unfinished" if results < len(message.tool_calls) else "compressible"
            groups.append(Group(i, end, keep))
            i = end
            continue
        if message.role == "system":
            keep = "system"
        elif options.pinned(message):
            keep = "pinned"
        else:
            keep = "compressible"
        groups.append(Group(i, i + 1, keep))
        i += 1
    return groups

After grouping and marking the keepers, the remaining consecutive groups form the compressible region; that region is then split at user messages — a user message opens a new piece of work, so splitting by work keeps a summary from blending two things into one sentence.

How to compress: have the model write a structured summary

Only now does the model take the stage, and its hands must be tied first.

A free-form summary reads smoothly and is lethal: it turns "divide has already been changed to throw" into "discussed how to fix the divide-by-zero problem" — an action became a topic. Next turn the model does not know it acted, so it acts again; and this time old_string no longer matches, so it starts suspecting someone else changed the file.

So the prompt fixes four headings and the model only fills in content:

Confirmed facts:
Changes already made:
Unfinished items:
Key paths and commands:

Their division: facts are conclusions that can be cited directly, changes are actions that must not be redone, unfinished items are the entry point for the next step, and paths and commands are the keys for re-fetching originals — that last line is what makes a compressed tool result genuinely re-fetchable. The prompt also says "content must come from the original; write none rather than inventing": once a summary speculates, the rest of the conversation rests on a sentence nobody said.

A fixed format makes it checkable, and that is today's most important safeguard:

Put each summary back in place, one message replacing a segment. Day eight's groundwork pays off here: injected material was its own message and never spliced into the user's sentence, so compaction can drop the injection and keep the words.

Finally, add a line to the system prompt: "the conversation history may have been compacted; messages starting with this phrase are summaries." Without it the model treats a summary as something the user just said, or keeps asking which file you meant.

How to verify: use early facts as probes

How do you know nothing was lost? "Looks fine" is not an answer — compaction fails silently: the model will not say "I lost a fact," it will confidently carry on with a wrong or invented one, and you notice several turns later from a strange result.

The verifiable method is probes: before compacting, record a few facts that exist only in the early context, and look for them afterwards.

The criterion is the key. The criterion is whether the fact is still in the context, not whether the model answers correctly. The latter is unreliable: a correct answer may be a lucky guess and a wrong one this turn's bad luck. Whether that sentence is in the context is deterministic, reproducible, and the only thing compaction can be held responsible for.

The self-test makes this a controlled experiment, because only a control proves anything:

v compaction: 14 messages -> 7, estimate 1368 -> 571 tokens, both summaries carry all four headings
v both probes are still findable in the compacted context: pnpm build:calc, divide by zero
v switching to "keep only the last three groups": the estimate drops to 358 tokens, but the fact "pnpm build:calc" is gone

The same session, the same keep list, with only "write a summary" replaced by "just discard": it saves even more tokens, and the early fact is gone. That is the entire difference between compaction and truncation, and what that extra request buys.

Two reminders. One: probes are best pinned by the user — sentences like "remember: the build command is..." are natural probes. Two: real mode deserves a second layer that actually asks the probe questions. The two layers mean different things: the first verifies compaction, the second verifies the model, and only the first is something you can fix.

Source Reading

Hands-On Lab

Today leaves five exercises, four of which are "looks simpler, is worse" traps: reporting coefficients without calibrating, one message per group (producing tool results with no matching call), triggering by turn count, and using a summary as soon as it arrives. The starter passes six of fifteen unmodified, all offline.

1. Implement the two-coefficient estimator and least-squares calibration, see the line "error 55.6% down to 2.6%," and confirm an all-ASCII sample is judged unsolvable.
2. Change triggering to proportion, and confirm with two histories of fourteen messages each that "the long one triggers and the short one does not."
3. Implement grouping and the keep list: assistant messages bound to their tool results, with system instructions, the pinned checklist, unfinished calls and the last three groups excluded from the compressible region.
4. Implement segment summaries with a format safeguard: one missing heading abandons the whole compaction, touching not one byte.
5. Run MOCK=1 SELFTEST=1 pnpm start to see 15/15 passed, then use /compact in the REPL and check the two probes; the self-test's numbers are reproducible, while turns that run real tests in the REPL fluctuate by tens of tokens.

Acceptance is five ticks: the self-test prints 15/15 passed; calibration error drops markedly and a uniform sample refuses a forced solution; two histories of equal length differ in triggering; after compaction the message count and estimate both drop, summaries carry all four headings and no orphaned tool results remain; and both probes survive compaction while the truncation variant loses one fact.

Interview Questions

Today's three questions test engineering judgment about tokens and compaction, not "what is a context window":

1. Without a tokenizer, how do you estimate tokens? How large is the error, and when can estimation not be used?
2. Which message categories must compaction keep? Which are safest to compress?
3. How do you prove a compaction lost no key information?

Full prompts, analyses and key points are in this course's day-twelve question bank. Question three discriminates most — most people get as far as "let the model judge," and few can state that the criterion is whether the context contains it, not whether the model answers correctly.

Checklist and Tomorrow

• I can state the division between estimation, real usage and exact counting, and why this course adds no tokenizer
• I can hand-write the two-coefficient least-squares calibration and explain why a near-zero determinant must not be forced
• I can give the reason proportion beats turn count, and why the trigger does not sit at ninety percent
• I can draw the seven-path budget table and state what fixed allowances really buy
• I know compaction priority follows "can it be re-fetched," not recency
• I can recite the four keep-list entries and the hard rule that an assistant message and its tool results are inseparable
• I can explain why summaries are structured, why a failed format abandons everything, and what the probe criterion is

Tomorrow is D13, "Ask Before Acting: a Structured Question Tool, Read-Only Exploration Mode, and Plan Approval." Today solved what to do when the desk is full; tomorrow solves putting less on the desk in the first place: ask what to change, explore read-only, produce an approvable plan, and only then act. Compaction comes before plan mode because plan mode lengthens the context considerably — it adds question, exploration and plan segments, all three of which are the least appropriate to compress. Settle "what to compress and what to keep" today so those three segments have somewhere to sit tomorrow.