Reference Injection: Parsing @ Files, Directories, URLs, and Images, and Accounting for What Got Injected

Parsing the reference syntax: escaping, spaces, wildcards and missing paths

The syntax has one symbol: @ followed by a path, a wildcard or a URL. Five lines of regex write it — and then four classes of bug arrive within a week, because it handles what people type casually.

Four cases, none skippable:

1. An email address is not a reference. In eric@example.com the character before @ is a letter. The criterion is "@ must be at the start or right after whitespace," not "the text contains an @." Without it, every email address a user writes sends the Agent to read a nonexistent path and report an error.
2. Escaping must work. \@src means I really meant those characters, do not read a file.
3. Paths with spaces must be quotable: @"src/my file.js". An unclosed quote counts as unfinished; do not guess.
4. Punctuation sticks to the end, in two kinds. CJK punctuation is a hard terminator: in a Chinese sentence the full stop must end parsing, because paths never contain it. But an ASCII dot cannot be a terminator — calc.js contains one — so it is only stripped when trailing. A slash is on neither list; it marks a directory and must stay.

/** Hard terminators: CJK punctuation, absent from paths and written flush against text */
const TERMINATORS = new Set(['。', '，', '、', '；', '：', '！', '？', '）', '」'])
/** Stripped only when trailing. An ASCII dot cannot terminate: calc.js contains one */
const TRAILING = new Set(['.', ',', ';', ':', '!', '?', ')', ']', '}', '"', "'"])
 
export function parseReferences(text: string): Reference[] {
  const refs: Reference[] = []
  for (let i = 0; i < text.length; i += 1) {
    if (text[i] !== '@') continue
    if (text[i - 1] === '\\') continue // escaped
    // This line is "an email is not a reference": @ must start the text or follow whitespace
    if (i > 0 && !/\s/.test(text[i - 1] as string)) continue
 
    const parsed = readTarget(text, i + 1)
    if (!parsed) continue
    refs.push({ kind: classify(parsed.target), target: parsed.target, start: i, end: parsed.end })
    i = parsed.end - 1
  }
  return refs
}

TERMINATORS = set("。，、；：！？）」")
TRAILING = set(".,;:!?)]}\"'")
 
 
def parse_references(text: str) -> list[Reference]:
    """@ must start the text or follow whitespace, else it belongs to an email or a handle."""
    refs: list[Reference] = []
    i = 0
    while i < len(text):
        if text[i] != "@" or (i and text[i - 1] == "\\"):
            i += 1
            continue
        if i and not text[i - 1].isspace():
            i += 1
            continue
        parsed = read_target(text, i + 1)
        if parsed is None:
            i += 1
            continue
        target, end = parsed
        refs.append(Reference(kind=classify(target), target=target, start=i, end=end))
        i = end
    return refs

The parse result keeps each reference's position in the original text. Why: the user's own words are not altered by one character. The @src/calc.js stays in their sentence — it is a pointer for the model, which cites it when answering, and the user can match it when scrolling back.

As for a missing path: do not crash, do not inject, print one plain sentence, and say where you looked. The lab's message reads "not found (no such path under work/repo or the current directory)." Why name the places searched — because references have two roots: the "this file" in the user's head may be relative to the repository the Agent is editing, or to the directory where they started mca. Search both, but print which root actually resolved; guessing right without saying so leaves them guessing next time.

What a directory reference expands into: a file tree or file contents

This is today's first real design decision, and an easy one to get wrong.

What should @src/ inject? Stuffing in the contents of every file under it sounds "more complete." The starter does exactly that, and one run shows the accounting line jumping to thousands of characters — while the file the user actually cared about may have been squeezed out by the budget.

This course's rule: a directory gives a file tree, not contents.

The reason in one sentence: a directory is a scope, not material. Writing @src/ says "the relevant thing is somewhere in here," not "read every character of this area." Give it a list and it picks the one or two to read — and it already has read_file.

you > look at @src/ and @work/repo/README.md and @https://example.com/doc
Injected 3 sources, 345 characters total (about 203 tokens, 1% of the reference budget)
  v src/  72 chars - about 44 tokens
  v work/repo/README.md  105 chars - about 59 tokens
  v https://example.com/doc  168 chars - about 100 tokens

One more rule alongside: a wildcard (@src/**) is a third form, expanding into many file sources. Those must be distinguished from the ones the user named individually — the tiering matters in the pruning section below.

The file injection format has a small decision too: include line numbers, in exactly the format read_file outputs. The model should not learn two "what a file looks like" formats; seeing line numbers it speaks in line numbers, and edit_file and read_file use the same numbering.

Three things about URL references: fetching, text extraction, graceful failure

URLs are the most troublesome of the three kinds, because they involve a party you do not control. Each of the three things has a discipline.

Fetching needs a timeout, a size cap and a content-type check, all three. Without a timeout, a turn hangs on an unresponsive server; without a size cap, a page of tens of megabytes is in memory before you compute the budget (so cap while reading, not after); without a content-type check, you inject the bytes of a PDF or an image as text.

Text extraction only needs to be usable; do not add a third-party library. The goal is not perfect structural fidelity but "strip navigation, scripts and styles, keep readable paragraphs." Three steps:

export function extractText(html: string): string {
  // Order matters: delete script and style blocks entirely first,
  // or step two turns tags into whitespace and script code becomes body text
  const withoutBlocks = html
    .replace(/<script[\s\S]*?<\/script>/gi, ' ')
    .replace(/<style[\s\S]*?<\/style>/gi, ' ')
  const withBreaks = withoutBlocks
    // Block-level tags become newlines so some sense of paragraphs survives
    .replace(/<\/(p|div|li|h[1-6]|tr)>/gi, '\n')
    .replace(/<[^>]+>/g, ' ')
  return collapse(decodeEntities(withBreaks))
}

BLOCK_END = re.compile(r"</(p|div|li|h[1-6]|tr)>", re.I)
TAG = re.compile(r"<[^>]+>")
 
 
def extract_text(html: str) -> str:
    """Delete script and style blocks first, then tags - reversed, code becomes body text."""
    body = re.sub(r"<script.*?</script>", " ", html, flags=re.S | re.I)
    body = re.sub(r"<style.*?</style>", " ", body, flags=re.S | re.I)
    body = BLOCK_END.sub("\n", body)
    body = TAG.sub(" ", body)
    return collapse(unescape(body))

Order is this code's only subtlety: delete blocks first, then tags. Reversed, code inside script becomes body text — and the model takes it seriously and quotes your analytics snippet in its answer.

Failure degrades to a plain sentence, not a crashed turn. Cannot fetch: say so and why. Fetched but no body extractable (usually a client-rendered page): say so and suggest pasting the content directly. Do not guess: injecting an empty shell is worse than injecting nothing, because the model believes it has read the page.

An offline note: under MOCK=1 the lab sends no network requests, and URL references use a built-in sample page. The extraction logic runs the same code, so the path is verifiable on a machine with no network.

Binary and oversized files: refusing is more honest than truncating

Two kinds of bad material, both handled by refusal, not "best effort."

Binary is easy: reading a null byte basically means binary, and injecting garbage wastes budget (the criterion matches day three's read_file).

Oversized files deserve a sentence, because "just truncate it" sounds too natural.

It is not fine. Truncate a log file and the model gets a head and a tail while the critical error line sits exactly in the removed middle — and it cannot tell it was misled, so it concludes confidently from incomplete material. Refusing with a next step ("read it in slices with read_file and an offset, or reference just one section") gives it a chance to obtain the part it actually needs.

Then why is truncation allowed at the per-source cap? Because that file already passed the size check, its magnitude is controlled (half the budget), and the cut point clearly states how many characters were removed and how to read the rest. The dividing line is "slightly over" versus "an order of magnitude over."

The lab prints one line per bad case, all marked with an x and none injected:

Injected 0 sources, 0 characters total (about 0 tokens, 0% of the reference budget)
  x src/nope.js  not found (no such path under work/repo or the current directory)
  x work/fixtures/logo.png  looks like a binary file (4 bytes), not injected
  x work/fixtures/huge.txt  too large (254 KB, cap 195 KB), not injected. Use read_file with an offset
  x src/*.rs  this wildcard matched no files

Report the injected volume: characters, estimated tokens, percentage of budget

Now the discipline of the day.

Everything in the context costs money and squeezes something else out. And injection has a particular property: the user cannot see it. They wrote three @s and see one line on screen, while twenty thousand characters may have gone in. They discover it when things are slow or the answer is wildly off.

So: every injection path reports its own numbers — characters, estimated tokens, and what percentage of that path's budget was used. The lab's line — Injected 1 source, 252 characters total (about 86 tokens, 1% of the reference budget) — is the whole thing.

Three implementation decisions:

• The accounting goes to the terminal, not to the model. These numbers help a human decide; sending them to the model just wastes budget — the same reasoning as day three's "tool result meta is not fed back."
• Tokens are estimated, not counted. The lab uses split coefficients (one token per CJK character, one per four characters otherwise). It is definitely inaccurate; real counting needs a tokenizer, and all we want here is a magnitude that makes the line meaningful. Day twelve calibrates the coefficients against real usage returned by the gateway.
• The budget covers only this path. How the total is divided between the system prompt, instruction files, memory, history and references is day twelve's subject. Today only produces this path's ledger, so day twelve has something to divide.

The pruning rules must be explainable, because they really do discard things. This course's three, in priority order:

export function applyBudget(sources: Source[], budget = REF_BUDGET_CHARS): BudgetResult {
  const singleCap = Math.floor(budget * 0.5)
 
  // 1. Each source obeys the per-source cap first: over it, cut from the middle,
  //    keeping both ends and stating how much was removed
  const staged = sources.map((source) =>
    source.content.length <= singleCap ? measure(source) : measure(truncate(source, singleCap))
  )
 
  // 2. Still over the total: drop, but only derived sources, starting from the last.
  //    Tier-one sources were named by the user; truncate rather than drop them - silently
  //    dropping a named file is the worst behavior, since the user assumes it was read
  let total = staged.reduce((sum, s) => sum + s.chars, 0)
  for (let i = staged.length - 1; i >= 0 && total > budget; i -= 1) {
    const source = staged[i] as Measured
    if (source.tier !== 'derived') continue
    total -= source.chars
    staged[i] = { ...source, content: '', chars: 0, dropped: true }
  }
  return summarize(staged, budget)
}

def apply_budget(sources: list[Source], budget: int = REF_BUDGET_CHARS) -> BudgetResult:
    """Tier-one sources are truncated rather than dropped; drop derived ones from the end."""
    single_cap = budget // 2
    staged = [
        measure(s if len(s.content) <= single_cap else truncate(s, single_cap)) for s in sources
    ]
 
    total = sum(s.chars for s in staged)
    for i in reversed(range(len(staged))):
        if total <= budget:
            break
        if staged[i].tier != "derived":
            continue
        total -= staged[i].chars
        staged[i] = replace(staged[i], content="", chars=0, dropped=True)
    return summarize(staged, budget)

One judgment underlies all three rules: what the user named explicitly has the highest priority. That is where the tiering comes from — files and URLs written out individually are tier one, those brought in by a wildcard or directory are tier two. And drops must be reported (the accounting line ends with "N more sources dropped for lack of budget"); silent discarding is today's red line.

One structural decision that pays interest on day twelve: injected content is its own message, not spliced into the user's sentence. Splicing loses three things — the model stops citing the name the user wrote, the user cannot see their own words when scrolling back, and compaction can only drop the whole thing (whereas ideally you drop the injection and keep the words).

Proactive injection versus letting the model search: when to skip the step

Finally, the unavoidable question: since it has glob, grep and read_file, why @ references at all?

Because the cost structures differ:

The criterion is the last row: inject when the user knows which file matters; let the model search when the user is looking too.

Conversely, two situations where injection is wrong: one, you are not actually sure which file matters and pasted five in on a hunch — that is not pointing the way, that is stuffing noise into the context, possibly squeezing out the useful one; two, the material changes during the conversation (a file currently being edited), where an injection is a snapshot while a tool read is the present — there it must read for itself.

A word on images in today's title: referencing a screenshot is no different at the parsing layer (one more form), but it must travel the multimodal message path, an entirely different matter from text injection. That is day nineteen's subject, and today's implementation refuses images down the "binary file" path — not laziness, but not pretending to a capability that is not in place.

Source Reading

Hands-On Lab

Today leaves five exercises, three of which are "looks more thorough, is actually worse" traps: injecting directory contents, truncating oversized files instead of refusing, and dropping from the front when over budget (which discards sources the user named). The starter passes seven of twelve unmodified.

No child processes and no network today: every step of the @ path is directly callable, and web fetching uses a built-in sample page under MOCK=1. The binary and oversized fixtures are created by the self-test under work/fixtures.

1. Implement the reference parser recognizing files, directories and URLs while keeping original positions; run /refs try @**/*.js and eric@example.com and confirm the email was not treated as a reference.
2. Implement the expander: files with line numbers, directories as a listing only, wildcards expanding into several tier-two sources; run @src/ and confirm not one byte of file content is injected.
3. Add the injection budget and pruning rules, preserving explicitly named sources first and reporting anything dropped.
4. Print this turn's source list with characters, estimated tokens and percentage of budget; give one plain sentence for each of the four bad cases (missing, binary, oversized, wildcard with no match).
5. Run the self-test: MOCK=1 SELFTEST=1 pnpm start should print 12/12 passed. Parse results, injected characters, estimated tokens and the pruned total are reproducible; the session id and real web content are not.

Acceptance is five ticks: the self-test prints 12/12 passed; all three reference kinds parse and show volume and budget percentage; @src/ injects a listing only; the four bad cases each give a plain sentence and inject nothing; and once the material is in place that turn reaches a conclusion with zero tool calls, with the message array holding one injection message plus the user's own words.

Interview Questions

Today's three questions test governance judgment over context injection, not "how to read a file":

1. Implementing file reference injection, how do you handle directories, binaries and oversized files?
2. When injected context exceeds the budget, how do you prune? Should the pruning rules be visible to the user?
3. When should content be pushed into the context proactively, and when should the model fetch it with a tool?

Full bilingual prompts, analyses and key points are in this course's day-eight question bank. Question one is the easiest to answer shallowly — three kinds of material map to three different handlings (a listing, refusal, refusal with a next step), and few can explain why refusing is more honest than truncating.

Checklist and Tomorrow

• I can name the four cases @ parsing must handle, especially the criterion for "an email is not a reference"
• I know why a directory gives only a file tree, and the difference between scope and material
• I can name the three requirements for fetching a page, and why extraction deletes blocks before tags
• I can explain why refusing an oversized file is more honest than truncating, and when truncation is acceptable
• I can state the three pruning rules and why silent discarding is a red line
• I know why injected content is its own message rather than spliced into the user's sentence
• I can state the criterion between proactive injection and model-driven search, plus the two cases where injection is wrong

Tomorrow is D9, "Project Instruction Files: Three-Layer Loading, Import Expansion, and the System Prompt Merge Order." Today solved "what to look at this once"; tomorrow solves "what it should know every time" — your code style, how tests are run, which directories not to touch. Those should not be repeated in every conversation; they belong in a file it reads itself. References come before instruction files because references are one-off and controllable while instruction files are permanent and override each other: use the simple path to establish the accounting discipline, so tomorrow's three-layer merge has a ledger to reconcile against.