The Flattening Tax: Why LLMs Shouldn't Read Raw JSON

In early software engineering, web browsers and microservices standardized on JSON because human readability and dynamic schema evolution outweighed network byte overhead. When language models emerged, engineering pipelines unthinkingly adopted the exact same convention. Today, production applications routinely serialize multi-megabyte relational rows, API payloads, and state dumps as raw JSON and feed them directly into self-attention layers.

The architectural consequence is severe. Self-attention scales quadratically with sequence length. When an LLM ingests a JSON array containing hundreds of objects, between 60% and 80% of the active context window is occupied entirely by syntax: recurring quotation marks, curly braces, colons, brackets, and repeated column identifiers that appear identically across every row. The transformer attention heads are forced to compute pairwise attention scores over typographical delimiters that carry exactly zero factual entropy.

The industry's conventional response has been prompt compression: using smaller auxiliary models or semantic summarizers to delete words or truncate text. This is a dangerous misapplication. Semantic compressors discard tokens heuristically, introducing non-zero hallucination risks, dropping critical numerical bounds, and breaking foreign key associations. You cannot safely compress factual records by guessing which fields a downstream agent might need.

Contex approaches this as a compilation problem rather than a compression heuristic. When LLVM compiles high-level C or Swift into machine code, it does not summarize the developer's intent; it maps language abstractions into an optimal, typed intermediate representation (IR). By parsing structured payloads into a canonical, schema-typed intermediate representation before tokenization, we eliminate the structural syntax tax while guaranteeing bit-for-bit semantic equivalence.

When tested across standard frontier models including Claude 3.5 Sonnet and GPT-4o, models operating on compiled canonical IR achieve 100% extraction precision and identical mathematical reasoning accuracy, while eliminating up to 80% of prompt token prefill overhead. LLMs do not need human-friendly web syntax; they need structured semantic clarity.