A transformer-only inference architecture that serves large language models 3–5× cheaper than GPUs — using commodity memory, a low-latency NRZ fabric, and no HBM, no CoWoS, and no leading-edge process node.
A GPU decode step reads the model's active weights and every active user's KV-cache from memory, then performs comparatively little arithmetic. Despite reticle-sized logic dies and 2.5D-integrated HBM, decode runs far below peak FLOPs — silicon and power sit idle waiting on memory. The reflex fix is more bandwidth (more HBM, or SRAM), which is expensive, supply-constrained (HBM and CoWoS are sold out and concentrated among a handful of buyers), and solves the wrong problem.
Throughput inference serving is the one AI workload where the three most expensive design choices buy nothing:
Generalist accelerators cannot decline these costs because they must also serve training and latency-critical workloads. A specialist can.
Remote memory is only as fast as the link that reaches it, so the link must be balanced to the memory behind it. Because activations are tiny, the requirement is low latency, not high bandwidth. We use plain 32 GT/s NRZ SerDes (PCIe-class, standard licensable IP — no PAM4, no FEC, no switch) integrated on-die, arranged as a mesh: a small full-mesh pod for the latency-critical domain, with a scalable topology between pods. A direct NRZ hop moves an activation in ~100 ns; switched PAM4 fabrics pay ~600 ns of hardware latency plus microseconds of collective-software overhead to move the same kilobytes.
Each chip carries commodity LPDDR (≈1 TB/s per chip across many channels, with large capacity per package) and holds a shard of the model plus the KV-cache for the layers it computes. Critically, the large data (weights and KV) stays local to the chip that computes over it; only activations traverse the fabric. This is the opposite of a shared cross-chip memory pool — and it is precisely what allows the cheap wire.
Sharding cuts both bandwidth and FLOPs per chip, so compute is sized to the memory it is fed, not maximized in isolation. The result is a transformer-only datapath — systolic matmul, attention, feed-forward, and a top-k mixture-of-experts router — with just enough MACs to saturate the memory stream, and the remaining die budget spent on I/O ports and SRAM.
The inter-chip traffic per link equals tokens/second × activation size. For a hidden state of a few kilobytes per token, a single NRZ lane runs at low-single-digit-percent utilization at hundreds of concurrent users, and the load is independent of chip count and model size — it scales only with users and activation width. The design invariant is to keep the KV-cache local; violating it (a shared-memory pool) is what forces an expensive fabric. Latency, not bandwidth, is the lever for per-user speed, which a short-reach NRZ mesh optimizes directly.
A right-sized rack for ~100 concurrent users on a GLM-class mixture-of-experts model — provisioned for realistic average context and burstable to very long context via dynamic KV allocation — is built from tens of small, cheap chips. Because the die is small and efficient on a mature node, and the memory is commodity, per-rack cost lands well below an equivalent GPU deployment while drawing a fraction of the power. The customer pays once and owns the hardware, their models, and their data, with no recurring per-token bill.
| Design axis | Conventional GPU | Tarrasque |
|---|---|---|
| Node | Leading edge (3–4 nm) | Mature (12/6 nm) |
| Memory | HBM + CoWoS (gated) | Commodity LPDDR (ungated) |
| Interconnect | PAM4 + switch (high latency) | Direct NRZ mesh (low latency) |
| Optimized for | Training + latency + generality | Throughput inference serving |
| Deployment | Rented, per-token, off-prem | Owned, pay-once, on-prem |
The direction is corroborated by the field: leading independent architects have publicly concluded that HBM is a losing basis against incumbents for inference; four of the five most prominent AI-chip startups deliberately avoid HBM and CoWoS; and the dominant GPU vendor's newest long-context inference product uses commodity GDDR rather than HBM for the same cost, power, and packaging reasons argued here.
The full transformer/MoE compute datapath, the NRZ chip-to-chip fabric, and a memory-agnostic streaming interface are implemented and verified in simulation against numerical golden models, including a two-board pipeline passing a token end-to-end. The near-term path is an FPGA benchmark of the fabric and datapath at line rate, a low-cost multi-project-wafer test chip to validate the analog blocks, and first design-partner deployments — leading to a production part on a mature node with commodity memory.
Tarrasque Industries builds cheap, sovereign, on-premise AI inference. To join the waitlist or the design-partner program, contact hello@tarrasque.com.
This document describes work in progress and forward-looking architecture; figures are engineering estimates subject to silicon validation.