The Evolution of Memory-Centric Architectures for Large-Model Inference

Abstract

Reasoning models, long contexts and agents are shifting inference bottlenecks from peak compute toward frequent movement of weights, KV caches, expert parameters and intermediate state across memory tiers and interconnects. This article introduces a five-dimensional framework—workload, state object, storage hierarchy, data movement and service objective—and a joint capacity, bandwidth, latency and energy model for token generation. Prefill and decoding differ in arithmetic intensity, state lifetime and objectives, making a single optimal hardware configuration unlikely. Low-bit microscaling, KV compression and sparse retention reduce bytes, but their gains depend on conversion, accuracy recovery and reversibility. Disaggregation, global KV caches, CXL pools and near-memory compute elevate KV from device-local temporary data to independently schedulable state. Efficient architectures should combine value-aware placement, reversible compression, communication/computation overlap and heterogeneous chiplets, rather than merely add matrix units. Capacity examples, a token Roofline method and reproducible metrics support joint chip, compiler and serving-system design.

Transformers use self-attention to capture long-range dependencies and have enabled large language models to scale^[1]. Inference and training follow different performance regimes: training typically uses large matrix batches, whereas online inference combines compute-intensive prefill with low-batch autoregressive decoding and strong sequential dependencies^[2]. Reasoning models extend output sequences, while multi-turn agents repeatedly load system prompts, tool definitions, retrieved evidence and history^[3]. Fitting a model in accelerator memory is therefore not the same as running it efficiently. Ma and Patterson^[4]identify high-bandwidth flash, near-memory compute, 3D memory–logic stacking and low-latency interconnects as opportunities for inference hardware. All address compute growth outpacing memory capacity, bandwidth and movement efficiency. Rather than treating GPU, NPU, CXL, PIM and chiplets separately, this article follows token-generation state through SRAM, HBM, host DDR, CXL, NVMe and remote nodes, examining creation, residency, reuse, compression and migration to explain memory-centric architectures and their limits.

1. Technology evolution and scope

1.1 From compute-graph optimization to I/O-aware operators

Early Transformer optimization centered on parallel matrix multiplication, but standard attention constructs or repeatedly accesses an intermediate matrix of size $O(T^{2})$ , where $T$ is sequence length. FlashAttention tiles queries, keys and values to fit on-chip storage and uses online softmax to avoid materializing large intermediates in HBM, rewriting exact attention as an I/O-aware algorithm without changing its mathematical complexity^[5]. FlashAttention-2 improves block and warp work allocation, reducing non-matrix work and increasing parallelism^[6]; FlashAttention-3 adapts asynchronous execution, warp specialization and FP8 paths to Hopper-class hardware^[7]; FlashAttention-4 reorganizes pipelines, softmax scaling and work scheduling around asymmetric scaling of matrix units, shared memory and exponentiation units^[8]. Even with unchanged mathematics, optimal implementations depend on memory hierarchy, specialized-unit balance and instruction semantics. Operator optimization fundamentally changes how long data stays resident and how often it is accessed.

1.2 From continuous batching to paged state management

A second line of serving research combines request scheduling with KV management. Orca reschedules requests after each decoding iteration, forming dynamic batches that avoid request-level batching tails^[9]. PagedAttention maps logically contiguous KV to noncontiguous physical blocks, reducing reservation and fragmentation while enabling block sharing^[10]. vAttention uses GPU virtual memory to preserve contiguous virtual addresses and commit physical memory on demand without changing attention-kernel layouts^[11]. SGLang's RadixAttention organizes shared prefixes for tree-structured reuse of system prompts, retrieved documents and branching generation^[12]. Together, these approaches turn KV from an internal attention tensor into a runtime object with addresses, ownership, reference counts and eviction policies.

1.3 From device optimization to disaggregation

Prefill usually has higher arithmetic intensity and suits large GEMMs. Decoding repeatedly reads weights and historical KV, making bandwidth and tails critical. Sarathi-Serve uses chunked prefill to form more uniform mixed batches and improve utilization without blocking decoding for long periods^[13]. DistServe places prefill and decoding in separate GPU pools, planning resources and parallelism around TTFT and TPOT^[14]; Splitwise examines disaggregation through resource ratios and cost^[15]; Mooncake builds a global KV cache across GPUs, CPUs, DRAM, SSDs and network interfaces so compute and state nodes can scale independently^[16]. NanoFlow overlaps operators, requests and device pipelines at fine granularity, showing why one-level optimization cannot cover the full serving path^[17].

Figure 1. Evolution of memory technologies for LLM inference

1.4 Scope of this article

Memory-centric design does not mean moving all computation into memory or maximizing capacity. It organizes systems around state objects, explains bottlenecks through data movement and constrains architecture by service objectives. The scope includes dense, GQA, MoE and reasoning models and agent workflows, covering GPU/NPU runtimes, low-bit formats, KV management, CXL tiers, PIM/NDP, chiplets and evaluation. Training, update consistency and training communication appear only where technologies overlap with inference.

2. Inference workloads and the memory wall

2.1 Prefill–decode asymmetry

Let batch size be $B$, input length $T_{in}$, generated length $T_{out}$ and hidden dimension $d$. Prefill sends $BT_{in}$ tokens through matrix multiplication in parallel, amortizing weight reads and usually achieving high arithmetic intensity. Decoding generates few tokens per request per round, reducing work to small GEMMs or GEMVs and increasing the share of weight and KV reads. Each attention step compares the current query with historical keys and aggregates historical values, so traffic grows approximately linearly with retained context. Figure 2 relates phases, state and data paths.

Figure 2. Inference phases, state objects and data paths

End-to-end latency can be decomposed as:

Here, $T_{queue}$ is queuing latency, $T_{tool}$ is external-tool execution latency, and $T_{transfer}$ is state-transfer latency across tiers or nodes. Faster kernels do not guarantee a lower Equation (1), as queuing, migration and synchronization may offset gains. Larger batches amortize weights but may worsen TPOT and P99. Any claim of speed must specify workload distribution, service objectives and resources.

Table 1. Typical inference workloads and dominant bottlenecks

2.2 Different state lifecycles

Inference contains at least six state classes: large, read-mostly weights shared across requests; growing, request-bound KV with reusable prefixes; short-lived, bandwidth-intensive activations; routed, skewed MoE expert weights; prefixes, evidence and tool definitions whose reuse may outlast a session; and draft-model, verification and tool-result state whose value depends on future control flow.

Eviction based only on tensor size may discard expensive, likely-to-be-reused state while retaining large low-value objects. The residency value of state block $i$ can be defined as:

Here, $R_{i}$ is future reuse probability, $C_{i}$ is recomputation cost, $L_{i}$ is contribution to service objectives or task correctness, $M_{i}$ is migration cost, and $X_{i}$ is the opportunity cost of occupying fast memory. Equation (2) is a decision framework, not a fixed algorithm: it adds semantic value, recomputation cost and accuracy risk beyond LRU/LFU. For system prompts and tool descriptions, $R_{i}$ is high; for temporary reasoning on failed branches, $R_{i}$ may be low; for early code dependencies and exact formatting constraints, $L_{i}$ is high, so low attention scores alone do not justify deletion.

2.3 New pressure from reasoning models and agents

Longer reasoning permits more search and verification, but every additional token writes KV across layers and is reread later. DeepSeek-R1 demonstrates that reinforcement learning can induce long-chain reasoning^[3]. Systemically, output length becomes a task-dependent random variable. Planning capacity as fixed maximum output times concurrency either wastes capacity or causes preemption and swapping when difficult requests arrive together.

Agents add uncertain control flow. Tool calls pause GPU work but create prefetch windows; retries may reuse history or branch; multiple agents may share documents, plans and tools. State therefore needs a graph lifecycle with branching, merging, freezing and restoration, rather than simple allocation at request start and release at completion. Evaluation should measure whether an architecture exploits these distinctions, not only single-turn chat throughput.

3. A unified memory-centric framework

3.1 Five dimensions

Figure 3 presents workload, state object, storage hierarchy, data movement and service objective. Workloads determine lengths, concurrency and reuse; state defines sharing, loss tolerance and recomputation; tiers provide capacity, bandwidth, latency, energy and consistency; movement covers loading, prefetching, migration, compression, replication and near-memory execution; service objectives prevent average-throughput gains at the expense of accuracy or tails.

Figure 3. Five-dimensional framework and tiered state space

The framework explains apparent contradictions. Moving KV from HBM to DDR frees capacity but can sharply increase TPOT if every decode step reads it over PCIe. PIM attention reduces KV movement, but GPU projections and feed-forward work may introduce costly switching and layout conversion. Four-bit weights reduce bytes, but unsupported formats can make dequantization, scaling and outlier handling the bottleneck. Comparisons generalize only when all five dimensions are explicit.

3.2 Weight and KV capacity

For a model with $P$ parameters and weight bit width $b_{w}$ , ideal weight storage is:

Deployment additionally needs scales, zero points, sparse indices, shards, workspace and replicas. For standard KV caching, let layer count be $L$, batch size $B$, current sequence length $T$, KV-head count $H_{KV}$, head dimension $d_{h}$ and KV element bit width $b_{KV}$. Cache capacity is:

The factor of two represents keys and values. Equation (4) scales linearly with batch size, layers, sequence length and KV heads. MQA reduces $H_{KV}$ to one^[18], while GQA shares each KV group among multiple query heads^[19], directly reducing per-token storage. Multi-head latent attention compresses cacheable state into a low-dimensional latent representation and positional components^[20], changing the conventional head-based capacity calculation.

Table 2 gives three illustrative configurations, not specific commercial products: A is MHA with 32 layers, 32 KV heads and head dimension 128; B is GQA with 80 layers, eight KV heads and dimension 128; C has 60 layers caching a 576-dimensional latent vector per layer. Values derive solely from Equation (4) and the latent-cache formula, illustrating capacity magnitudes rather than device measurements.

Table 2. Modeled single-request KV capacity by attention architecture

Figure 4. Modeled KV capacity by architecture and precision

Bit-width reduction and structural compression multiply their benefits but change different things: structure reduces logical state dimensions, while quantization reduces bytes per element. They should not be conflated as one compression ratio. Structural changes generally need training or architecture adaptation; post-training quantization can introduce conversion costs and accuracy risks.

3.3 Token Roofline

Classical Roofline connects arithmetic intensity, peak compute and bandwidth. Inference also needs distinct weight, KV and network paths. Let one generated token require $F_{tok}$ effective operations, weight traffic $D_{W}$, KV traffic $D_{KV}$ and cross-device or network traffic $D_{N}$, with effective throughputs $P_{eff}$, $BW_{W}$, $BW_{KV}$ and $BW_{N}$. A lower bound on token latency is approximately:

Here, $T_{sched}$ and $T_{sync}$ denote scheduling and synchronization overhead. A maximum rather than a sum reflects ideal overlap; real systems add residual costs according to achieved overlap. Figure 5 places long-context decode attention at low intensity (A), quantized or near-memory decoding at B, and large-batch prefill GEMM at C. Quantization moves work right, bandwidth raises the slope, fusion removes reads/writes, and more matrix units raise only the compute ceiling. Extra FLOPS offer little while the workload remains bandwidth-bound.

Figure 5. Token Roofline illustration

Ignoring lower-order query and output traffic, each MHA/GQA KV read contributes approximately one multiply-add. With query-head count $H_{Q}$, KV-head count $H_{KV}$, attention arithmetic intensity is approximately:

Equation (6) shows that GQA and low-bit KV increase work per byte as well as reduce capacity. It omits indices, scales, masks, softmax and uncoalesced accesses. Sparse KV's theoretical savings may not become proportional speedups when indexing and random access are costly. Report actual bytes read, effective bandwidth utilization and kernel occupancy alongside compression ratios.

3.4 Energy and cost

Movement energy depends on memory tier. To avoid mixing incompatible process nodes, devices and measurements, we use a symbolic model. For state block $i$ in tier $k$, let access count be $n_{i}$, bytes $d_{i}$, access energy per byte $e_{k}$, migration energy $e_{m}(i,k)$ and compute energy $E_{comp}$. Request energy is:

Here, $π(i)$ represents placement. Cost also includes depreciation, idleness, memory occupancy and network resources. Cost per token can be expressed as $C_{tok}=C_{time}/N_{tok}+C_{energy}+C_{memory}+C_{network}$. Minimum energy per request and minimum cluster cost may differ. Moving cold KV to slower storage increases restoration energy but may improve expensive GPU utilization. Excessive prefetching can reduce latency while wasting bandwidth and energy.

4. Reducing required data movement

4.1 MQA, GQA and latent caches

Reducing KV dimensions is the most direct structural approach. MQA shares one key/value group across all query heads, reducing decoding storage and bandwidth^[18] but potentially limiting expressiveness. GQA balances query and KV head counts and can recover much of the quality when converted from MHA checkpoints^[19]. Multi-head latent attention uses low-rank compression and handles rotary-position components separately^[20]. All three change what must be retained at model-design time, benefiting runtimes and hardware but requiring training adaptation and possibly extra projection, decompression or custom kernels.

Structural compression should consider deployment topology. With tensor parallelism, KV-head count affects sharding and replication. When $H_{KV}$ is smaller than device count, uniform sharding may leave devices without independent heads, requiring replication or another parallelism scheme. Latent-cache savings disappear if runtimes still allocate MHA layouts. Runtimes must recognize attention variants rather than force all models into one template.

4.2 Low-bit weights, activations and KV

SmoothQuant transfers activation outliers into weights to support INT8 weights and activations^[21]; AWQ protects important weight channels based on activation distributions for efficient weight-only quantization^[22]; GPTQ minimizes blockwise error using approximate second-order information^[23]; QuaRot uses rotations to suppress outliers and regularize distributions for end-to-end 4-bit paths^[24]. Low-bit inference therefore combines distribution shaping, scale design, error compensation and kernels, rather than merely reducing bit counts.

FP4 and microscaling make format differences more important. MXFP4 commonly uses block-shared power-of-two scales; NVFP4 uses finer scaling and different encoding constraints. MR-GPTQ identifies problems with conventional outlier suppression in small NVFP4 groups and additional MXFP4 scale error, proposing format-aware rotations and kernels^[25]. AdaMX represents block and operand heterogeneity with limited metadata and corresponding accelerator decoding^[26]. SageAttention3 uses FP4 attention matrix multiplication while retaining higher-precision online softmax state^[27]. Compilers must therefore describe element format, block size, scale format, accumulation, outlier paths and target instructions explicitly, not simply accept a “4-bit” attribute.

KV differs fundamentally from weights: weights are known before deployment and can be calibrated offline; KV is generated online and varies by layer, head, token and task, placing quantization on the latency-critical path. KIVI uses asymmetric 2-bit KV quantization with different granularities for keys and values^[28]; KVQuant studies per-channel formats, nonuniform codebooks and outliers^[29]. Capacity roughly follows bit width, but end-to-end gains depend on quantized writes, dequantized reads, scale storage and whether attention kernels directly consume the representation.

4.3 Sparse retention, eviction and reversible compression

Sparse KV retains tokens expected to matter later. H2O retains consistently high-attention “heavy hitters” within a fixed budget^[30]; StreamingLLM combines initial attention sinks and a sliding window for streaming beyond the training window^[31]; SnapKV selects important positions from observation-window attention patterns^[32]; PyramidKV assigns layer-dependent budgets using differences in information density^[33]. These methods generally perform lossy retention. Similar average benchmark accuracy does not establish identical long-generation, code or tool-call trajectories.

VeriCache drafts with compressed KV and verifies against full KV, seeking acceleration while preserving the full-cache output^[34]. Reversibility permits high-precision full state in slower storage and compressed state in HBM for most decoding, with less frequent restoration and verification. Combining compression, speculation and tiered storage offers more controllable correctness–performance tradeoffs than permanent deletion. Evaluations should separately report semantic accuracy, exact output agreement, tool success and structured-format correctness, rather than mask functional errors with perplexity alone.

4.4 Limits of compression benefits

Four limits matter. Metadata and uncoalesced reads prevent compression ratios from equaling bandwidth savings. KV compression may improve concurrency without reducing token latency when weights dominate. Small or irregular quantized/sparse kernels can lower matrix utilization. Lossy errors may accumulate with output length and control flow. Experiments should include uncompressed, capacity-bound, bandwidth-bound and mixed cases, separating increased residency from faster individual requests through ablation.

5. Improving locality and runtime utilization

5.1 Continuing evolution of I/O-aware attention

FlashAttention's central contribution is incorporating memory traffic into algorithm design. The first generation reduces HBM traffic through tiling and recomputation^[5]; the second reduces inter-thread communication and non-matrix work through partitioning^[6]; the third overlaps tensor cores and softmax with asynchronous copies and warp specialization^[7]; the fourth reorganizes pipelines and uses software approximations as matrix throughput grows faster than shared memory, exponentiation and general ALUs^[8]. The same applies to NPUs and custom chips: changing hardware moves bottlenecks, so compilers must reconsider tiling, prefetching, accumulation and synchronization rather than port old kernels unchanged.

Attention kernels should expose contracts for KV layout, quantization and scale granularity, paging/sparsity/variable-length metadata, on-chip storage and double buffering, and synchronization with external DMA or PIM. Without these, systems connect components through copies and conversions, producing local optima but weaker end-to-end results.

5.2 Paging, virtual memory and physical continuity

PagedAttention divides KV into fixed blocks and maps logical blocks to physical pages, reducing fragmentation from maximum-length reservations^[10]. It also supports copy-on-write and shared prefixes, but kernels must read page tables and handle discontinuity. vAttention preserves contiguous virtual addresses and standard kernels while allocating physical pages through GPU virtual memory^[11]. Neither is universally superior; page size, mapping cost, TLB reach, kernel compatibility and sharing granularity determine the tradeoff.

Small pages reduce fragmentation and copy amplification but increase page-table, TLB and scheduling overhead. Large pages improve sequential DMA and remote transfer but may move unused tokens. A dual-granularity design could retain small logical references while merging neighboring hot pages into large transfers and arranging PIM data by bank and row buffer. This requires coordination among translation, reference counts and compression beyond ordinary allocation.

5.3 Prefix caching and structured reuse

RadixAttention maps prefix tokens into radix-tree nodes so requests can share their longest common prefix^[12], benefiting system prompts, few-shot examples, RAG and tree search. Mooncake extends prefix KV into a distributed cache^[16]. Reuse is more than a hit/miss decision: model versions, RoPE scaling, formats, tensor parallelism and layouts may make otherwise identical prefixes incompatible.

KV needs an executable-state descriptor covering model and weight versions, tokenizer, positional parameters, layer range, head layout, precision, page size and topology. Text hashes alone are insufficient for cross-node keys. Content-addressed and position-dependent identities should remain distinct, with correction interfaces for future cross-position reuse, fragment composition and CacheBlend-like methods.

5.4 Scheduling, batching and overlap

Iteration-level batching admits new requests and removes completed ones after each decode step^[9]. Chunked prefill interleaves long inputs with decoding to prevent one request blocking the batch^[13]. Disaggregated scheduling also selects prefill and decode instances, transfer paths and start times. Request scheduling, memory allocation, congestion and cache hits must share a control loop.

Table 3 compares memory effects. Techniques optimize different terms and may conflict: tiny pages aid eviction but hurt SSD I/O; chunking reduces decode interference but adds KV segments and scheduling; disaggregation removes compute interference but adds network transfers. Report combined net gains rather than multiply speedups from unrelated papers.

Table 3. Memory behavior of inference techniques

6. Compute–state disaggregation and CXL tiers

6.1 Why separate prefill and decoding?

DistServe identifies different parallelism preferences and latency objectives, with interference when phases share resources^[14]. Splitwise assigns prefill to compute-rich devices and decoding to bandwidth- and latency-oriented devices based on overall cost^[15]. The value is independent scaling, batching and parallelism, not simply replacing one GPU type with two.

Disaggregation introduces KV handoff costs. Full transfers consume bandwidth and TTFT; shared-storage staging adds I/O and lookups. Ideally, destinations are selected before prefill and KV streams by layer or chunk as it is produced. For very long inputs, cold fragments may remain in shared memory while imminent hot blocks are prefetched into HBM.

6.2 Global KV caches and directories

Mooncake uses spare CPU, DRAM, SSD and NIC resources for disaggregated KV and selects reuse and transfer paths under SLO constraints^[16]. CacheGen encodes KV for more efficient network transfer^[35]; InfiniGen predicts important KV from current and previous layers for selective host-memory loading^[36]; FlexGen places tensors across GPU, CPU and disk with pipelining to support larger models^[37]. Tiered state is becoming a deliberate architectural resource rather than emergency swapping when accelerator memory runs out.

A global directory needs object keys, model compatibility, logical positions, physical replicas, references, popularity, formats, checksums, ownership and invalidation versions. Shared prefixes need concurrent references and copy-on-write; lossy state needs quality levels and recovery paths; agents need session DAGs and likely branches. Figure 6 shows the logical data plane.

Figure 6. Disaggregated prefill, decoding and global KV data plane

6.3 The role of CXL pools

CXL adds coherence and memory semantics over PCIe. CXL 3.1 strengthens fabric management, sharing and pooling^[38]; CXL 4.0 raises link rates to 128 GT/s and extends port aggregation, maintenance and RAS^[39]. Type-3 memory supplies host-addressable capacity, while switches and fabric managers make pools composable. Compared with SSD file access, it offers finer addressing and lower software overhead. Its bandwidth and latency still differ substantially from HBM; it is not transparently equivalent accelerator memory.

CXL can hold warm KV, shared prefixes, infrequent experts and full recovery state. Sequential, prefetchable transfers hide latency better than random per-step KV reads. Useful strategies keep hot windows in HBM, prefetch during tools or queuing, execute selection/attention near CXL and return aggregates, or compress links with device-side decompression. Its value combines pooled capacity with programmable data paths.

6.4 Networks, congestion and consistency

KV shares networks with computation traffic, weights and storage I/O. Scheduling only by idle GPUs can send many long-prefill transfers into one decode pool. Requests need compute, HBM, network and remote-memory credits, with P95/P99 transfer times influencing placement. Model, quantization, tokenizer and positional changes require precise replica invalidation rather than TTL alone.

KV contains internal representations of inputs. Unauthorized sharing, residual pages and side channels can leak information. Pools require isolation, encrypted transport, tenant keys, verifiable deletion and auditing; compression and PIM processing belong within the trust boundary. Ignoring isolation overhead can make experimental hit rates and bandwidth utilization unsuitable for production.

7. Near-memory computation, HBM-PIM and chiplets

7.1 Moving computation toward data

When Equation (5) is dominated by $D_{KV}/BW_{KV}$ , moving KV to a GPU is not the only option: dot products, pre-softmax reductions and value aggregation can execute near data. NeuPIMs sends GEMM to NPUs and GEMV-like decoding to PIM and schedules batches across both^[40]; PIM Is All You Need explores GPU-free inference with CXL-connected PIM^[41]; LLMCompass evaluates compute, bandwidth, area and cost before RTL implementation^[42]. PIM suits regular, low-reuse, bandwidth-bound operators with simple control flow, not indiscriminate relocation of the entire dynamic stack.

Three mismatches remain: prefill GEMM and decode GEMV need different resources; long KV may exceed one module and require sharding/reduction; sparsity and compression introduce indices, variable precision and irregular access. Hardware needs lightweight programmability, DMA, translation and hierarchical reduction, while runtimes decide which state should move.

7.2 Processing across memory tiers

PAM coordinates placement and attention across HBM-, DRAM- and SSD-PIM, keeping hot tokens in high-bandwidth tiers and cold tokens in larger tiers^[43]. TokenStack separates capacity and compute layers within HBM, using the logic die for DMA, translation, aggregation and migration-time quantization^[44]. These designs recognize that weights, activations and hot/cold KV need different physical organizations.

Memory-centric design jointly chooses compute and data locations. For state block $i$ with candidate tiers $K$ and operator $o$ executable on GPU, NPU or PIM at $q$, a multiobjective formulation is:

Constraints include capacity, bandwidth, PIM slots, SLOs and error. Instead of solving large integer programs online, generate candidate mappings offline from models and hardware, then switch using popularity, queues and context length. Hysteresis prevents short-lived demand changes causing repeated migration.

7.3 Chiplets and open interconnects

Advanced packaging separates compute, I/O, cache, HBM control and PIM logic. UCIe 3.0 supports 48 and 64 GT/s with improved calibration, management and larger systems^[45]. Beyond yield and IP reuse, chiplets tailor resource ratios: matrix-rich prefill dies, cache/reduction-rich decoding dies, I/O dies for CXL/RDMA/directories and HBM logic dies for transformations.

Chiplets turn on-chip communication into a package-network problem. MoE routing, all-reduce and KV shards compete for D2D links; clocks, power and heat affect sustained performance. Report bytes per token, utilization, hops and congestion tails, not just one die's peak throughput. Figure 7 is a conceptual analysis architecture, not a fabricated product.

Figure 7. Conceptual GPU–PIM–chiplet co-design

7.4 Hardware–software contracts and compilation

IR must express state and movement, including shareability, 4-bit storage with FP16 accumulation, and migratable but non-discardable objects. Descriptors should include lifecycle, reuse domain, error tolerance, recovery, predicted access, layout, topology and security domain. Compilers generate kernel variants and migration plans; runtimes select them using live queues.

Table 4 compares complementary roles: GPUs offer general computation and mature software, CXL provides capacity and composition, PIM reduces selected movement, and chiplets integrate differentiated resources. Effective systems combine tiers rather than rely on one universal chip.

Table 4. Capabilities and limits of memory-centric hardware

8. Engineering decisions and evaluation

8.1 From symptoms to technology choices

Identify the bottleneck before choosing technology. Capacity pressure with spare bandwidth favors structural/KV compression, paging and tiers. Weight-bound latency favors batching, weight quantization or effective bandwidth. KV-bound work favors low-bit KV, sparsity, PIM or windows. Prefill-bound TTFT favors FlashAttention, chunking and prefix reuse. Interference-driven P99 favors SLO classes, phase separation and network credits.

Table 5 provides a workload–technique matrix. High indicates usual direct benefits, medium depends on implementation and workload, and low means it is rarely the first choice. Recalibrate it using actual length distributions, reuse and hardware measurements.

Table 5. Workload–technique suitability

8.2 Common evaluation metrics

Speedups across different models, hardware, contexts, concurrency and SLOs are not directly comparable. Evaluate latency, throughput, memory, movement, energy/cost and quality. Goodput counts successful requests under TTFT, TPOT and error constraints. Reasoning and agents also need task success, tool correctness, structured-output validity and agreement with full-KV outputs.

Table 6. Recommended evaluation metrics

8.3 Minimum reproducibility requirements

Do not combine unrelated peak figures into a fictional unified experiment. Fix model, tokenizer, runtime, driver and kernel versions; publish traces or length, arrival and prefix-similarity distributions; separate cold-start and steady-state results; state whether reuse, transfers and quantization are included; report P50/P95/P99; define board, node or cluster power boundaries; and disclose simulator calibration, error and area/power sources.

Use four evidence levels: analysis, microbenchmarks, trace replay and end-to-end tests. Equations (3)–(8) identify candidates; GEMM, attention, DMA, CXL and network measurements provide effective parameters; traces test capacity and queues; applications validate SLOs and accuracy. Retain error ranges and avoid substituting theoretical bandwidth or kernel speed for request performance.

8.4 Architecture decision process

Measure weight, KV, activation and network traffic; identify workload and non-negotiable metrics; calculate capacity by context, concurrency and precision; select combinations that reduce bytes, improve locality, add tiers or move computation; define end-to-end cost with SLO, quality and security constraints; replay traces to test tails and congestion; then assess dedicated PIM or chiplets once software behavior is stable.

9. Open questions and research directions

9.1 From cache management to state execution

Research in 2026 increasingly treats KV as runtime control rather than storage alone^[46]. State may represent a fixed prefix, evidence, failed branch or pending tool node, with value determined by the task graph. Typed lifecycle protocols should support derivation, freezing, branching, merging, validation and reclamation, connecting caching, scheduling and agent orchestration in one state machine.

9.2 Adaptive, reversible precision tiers

Fixed INT4/FP4 cannot suit every layer, head and task. Formats should adapt to distributions, popularity, output length and risk while retaining recovery paths. Format-aware FP4^[25][26][27] and lossless cache verification^[34]provide complementary foundations: heterogeneous block representation and low-precision drafting with high-precision verification. One direction keeps hot HBM KV at 4 bits and full CXL KV at 8/16 bits, raising precision when acceptance falls or tool risk increases.

9.3 Cross-model and cross-position reuse

KV is usually bound to model weights and token positions. Model routing, draft–target pairs and upgrades create similar but incompatible state. Directions include transferable latent prefixes, position corrections, shared lower-layer representations and lightweight validation. These may reduce repeated prefill but introduce semantic, versioning and security challenges.

9.4 High-bandwidth flash and tiered PIM

High-bandwidth flash seeks much greater capacity through HBM-like parallel interfaces^[4], while tiered PIM places selection, reductions and attention subtasks near HBM, DRAM and SSD^[43][44]. Both need suitable layouts and control protocols: flash favors large sequential accesses, while attention uses dynamic indices; PIM modules need migration and reduction without a common programming model. Coarse prefetching, device-side selection and higher-tier verification may work better than emulating HBM.

9.5 Open standards, reliability and security

CXL and UCIe provide open interconnect foundations^[38][39][45], but model state lacks common cross-device formats. Candidates include KV metadata, compression negotiation, page migration, device operators, checksums and invalidation. Low voltage, low bits and 3D stacking raise reliability challenges; tenant sharing requires encryption, integrity, erasure and access evidence. RAS and security belong in per-token cost.

9.6 From peak performance to state efficiency

LLMCompass demonstrates faster design-space evaluation between cycle simulation and coarse Roofline^[42]. Future tools should include dynamic traces, tiered directories, CXL/network congestion, PIM migration and precision recovery. Metrics should measure SLO-compliant tokens per fast-memory unit, useful tokens per transferred byte and successful tasks per joule, encouraging state-efficient chips.

10. Conclusion

Inference is moving from compute-centric design toward state and movement. The five-dimensional framework and token Roofline explain why prefill, decoding, reasoning and agents differ and why peak FLOPS cannot determine end-to-end performance. Structural attention changes reduce state dimensions; low bits and sparsity reduce bytes; I/O-aware kernels, paging and prefixes improve locality; disaggregation, global KV and CXL expand state space; PIM and chiplets change physical distances.

Four conclusions follow: KV is typed, versioned, valuable and recoverable schedulable state; compression evaluation must include kernels, metadata, output length and functional correctness; CXL/PIM/chiplet gains depend on lifecycle, popularity, topology and congestion awareness; and future systems must predict value, place hot state, overlap movement and compute, and balance performance, energy, cost, reliability and security online.

Chips, compilers and serving systems should evolve together. Models expose reusable and compressible structure; compilers produce format- and tier-aware variants; runtimes maintain directories and SLO control; hardware provides composable memory, near-memory computation, translation and efficient D2D links. Memory-centric design evaluates computation within its complete data path.

Memory-centric architecture evolution for large-model inference

Abstract Reasoning-oriented large models, long-context applications and autonomous agents are shifting the dominant bottleneck of inference from peak arithmetic throughput to the movement of model weights, key-value caches, expert parameters and intermediate states across memory and interconnect hierarchies. To unify optimizations that are usually studied separately at the operator, runtime, memory-system and accelerator levels, this article develops a five-dimensional framework covering workload, state object, storage tier, data movement and service objective, together with a capacity-bandwidth-latency-energy model for token generation. The analysis shows that: ① prefill and decode exhibit fundamentally different arithmetic intensity, state lifetime and latency objectives; ② low-bit microscaling, KV compression and sparse retention reduce transferred bytes, but their gains depend on format-conversion overhead, accuracy recovery and reversibility; ③ prefill-decode disaggregation, global KV caches, CXL memory pools and near-memory processing elevate KV caches from device-local tensors to independently schedulable system state; and ④ future inference architectures should perform value-aware placement, reversible compression, communication-computation overlap and heterogeneous Chiplet coordination instead of merely adding matrix units. Model-based capacity cases, a Token Roofline method and reproducible evaluation criteria are provided to support joint design of accelerators, compilers and serving systems.

References

[1] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems. Long Beach: Curran Associates, 2017, 30: 5998-6008.

[2] POPE R, DOUGLAS S, CHOWDHERY A, et al. Efficiently scaling transformer inference[C]//Proceedings of Machine Learning and Systems. 2023, 5: 606-624.

[3] DEEPSEEK-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning[EB/OL]. 2025[2026-09-03]. arXiv:2501.12948.

[4] MA X, PATTERSON D. Challenges and research directions for large language model inference hardware[EB/OL]. 2026[2026-09-03]. arXiv:2601.05047.

[5] DAO T, FU D Y, ERMON S, et al. FlashAttention: Fast and memory-efficient exact attention with IO-awareness[C]//Advances in Neural Information Processing Systems. New Orleans: Curran Associates, 2022, 35: 16344-16359.

[6] DAO T. FlashAttention-2: Faster attention with better parallelism and work partitioning[C]//International Conference on Learning Representations. Vienna: OpenReview, 2024.

[7] SHAH J, BIKSHANDI G, ZHANG Y, et al. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision[EB/OL]. 2024[2026-09-03]. arXiv:2407.08608.

[8] ZADOURI T, HOEHNERBACH M, SHAH J, et al. FlashAttention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling[EB/OL]. 2026[2026-09-03]. arXiv:2603.05451.

[9] YU G I, JEONG J S, KIM G W, et al. Orca: A distributed serving system for Transformer-based generative models[C]//16th USENIX Symposium on Operating Systems Design and Implementation. Carlsbad: USENIX Association, 2022: 521-538.

[10] KWON W, LI Z, ZHU S, et al. Efficient memory management for large language model serving with PagedAttention[C]//Proceedings of the 29th ACM Symposium on Operating Systems Principles. Koblenz: ACM, 2023: 611-626.

[11] PRABHU R, NAYAK A, MOHAN J, et al. vAttention: Dynamic memory management for serving LLMs without PagedAttention[C]//Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. Rotterdam: ACM, 2025: 1133-1150.

[12] ZHENG L, YIN L, XIE Z, et al. SGLang: Efficient execution of structured language model programs[C]//Advances in Neural Information Processing Systems. Vancouver: Curran Associates, 2024, 37: 62557-62583.

[13] AGRAWAL A, PANWAR A, MOHAN J, et al. Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve[C]//18th USENIX Symposium on Operating Systems Design and Implementation. Santa Clara: USENIX Association, 2024: 117-134.

[14] ZHONG Y, LIU S, CHEN J, et al. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving[C]//18th USENIX Symposium on Operating Systems Design and Implementation. Santa Clara: USENIX Association, 2024: 193-210.

[15] PATEL P, CHOU C, CHARDON P, et al. Splitwise: Efficient generative LLM inference using phase splitting[C]//Proceedings of the 51st Annual International Symposium on Computer Architecture. Buenos Aires: IEEE, 2024: 118-132.

[16] QIN R, LI Z, HE W, et al. Mooncake: Trading more storage for less computation—A KVCache-centric architecture for serving LLM chatbot[C]//23rd USENIX Conference on File and Storage Technologies. Santa Clara: USENIX Association, 2025: 155-170.

[17] ZHU K, ZHAO Q, ZHAO H, et al. NanoFlow: Towards optimal large language model serving throughput[C]//19th USENIX Symposium on Operating Systems Design and Implementation. Boston: USENIX Association, 2025.

[18] SHAZEER N. Fast Transformer decoding: One write-head is all you need[EB/OL]. 2019[2026-09-03]. arXiv:1911.02150.

[19] AINSLIE J, LEE-THORP J, DE JONG M, et al. GQA: Training generalized multi-query transformer models from multi-head checkpoints[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, 2023: 4895-4901.

[20] DEEPSEEK-AI. DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model[EB/OL]. 2024[2026-09-03]. arXiv:2405.04434.

[21] XIAO G, LIN J, SEZNEC M, et al. SmoothQuant: Accurate and efficient post-training quantization for large language models[C]//Proceedings of the 40th International Conference on Machine Learning. Honolulu: PMLR, 2023: 38087-38099.

[22] LIN J, TANG J, TANG H, et al. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration[C]//Proceedings of Machine Learning and Systems. 2024, 6: 87-100.

[23] FRANTAR E, ASHKBOOS S, HOEFLER T, et al. GPTQ: Accurate post-training quantization for generative pre-trained transformers[C]//International Conference on Learning Representations. Kigali: OpenReview, 2023.

[24] ASHKBOOS S, MOHTASHAMI A, CROCI M L, et al. QuaRot: Outlier-free 4-bit inference in rotated LLMs[C]//Advances in Neural Information Processing Systems. Vancouver: Curran Associates, 2024, 37.

[25] EGIAZARIAN V, PANFEROV A, KUZNETSOV A, et al. Bridging the gap between promise and performance for microscaling FP4 quantization[EB/OL]. 2025[2026-09-03]. arXiv:2509.23202.

[26] LUO J, JIANG X, WEN T H, et al. Heterogeneity-aware microscaling for efficient low-bit LLM inference[EB/OL]. 2026[2026-09-03]. arXiv:2608.03867.

[27] LIN J, LUO M, ZHANG J, et al. SageAttention3: Fast attention on Blackwell GPUs via FP4 quantization[EB/OL]. 2025[2026-09-03]. arXiv:2505.11594.

[28] LIU Z, ZHAO C, FIELDER E, et al. KIVI: A tuning-free asymmetric 2bit quantization for KV cache[C]//Proceedings of the 41st International Conference on Machine Learning. Vienna: PMLR, 2024.

[29] HOOPER C, KIM S, MOHAN J, et al. KVQuant: Towards 10 million context length LLM inference with KV cache quantization[EB/OL]. 2024[2026-09-03]. arXiv:2401.18079.

[30] ZHANG Z, SHENG Y, ZHOU T, et al. H2O: Heavy-hitter oracle for efficient generative inference of large language models[C]//Advances in Neural Information Processing Systems. New Orleans: Curran Associates, 2023, 36: 34661-34710.

[31] XIAO G, TIAN Y, CHEN B, et al. Efficient streaming language models with attention sinks[C]//International Conference on Learning Representations. Vienna: OpenReview, 2024.

[32] LI Y, HUANG Y, YANG B, et al. SnapKV: LLM knows what you are looking for before generation[C]//Advances in Neural Information Processing Systems. Vancouver: Curran Associates, 2024, 37: 22947-22970.

[33] CAI Z, ZHANG Y, GAO B, et al. PyramidKV: Dynamic KV cache compression based on pyramidal information funneling[EB/OL]. 2024[2026-09-03]. arXiv:2406.02069.

[34] YAO J, SHEN S, DU K, et al. VeriCache: Turning lossy KV cache into lossless LLM inference[EB/OL]. 2026[2026-09-03]. arXiv:2605.17613.

[35] LIU Y, LI H, CHENG Y, et al. CacheGen: KV cache compression and streaming for fast large language model serving[C]//Proceedings of the ACM SIGCOMM 2024 Conference. Sydney: ACM, 2024: 38-56.

[36] LEE W, LEE J, SEO J, et al. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management[C]//18th USENIX Symposium on Operating Systems Design and Implementation. Santa Clara: USENIX Association, 2024: 155-172.

[37] SHENG Y, ZHENG L, YUAN B, et al. FlexGen: High-throughput generative inference of large language models with a single GPU[C]//Proceedings of the 40th International Conference on Machine Learning. Honolulu: PMLR, 2023: 31094-31116.

[38] CXL CONSORTIUM. Introducing Compute Express Link 3.1: Significant improvements in fabric connectivity, memory pooling and security[EB/OL]. 2023[2026-09-03].

[39] CXL CONSORTIUM. Compute Express Link 4.0 specification[EB/OL]. 2025[2026-09-03].

[40] HEO G, LEE S, CHO J, et al. NeuPIMs: NPU-PIM heterogeneous acceleration for batched LLM inferencing[C]//Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. La Jolla: ACM, 2024.

[41] GU Y, KHADEM A, UMESH S, et al. PIM is all you need: A CXL-enabled GPU-free system for large language model inference[C]//Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. Rotterdam: ACM, 2025: 862-881.

[42] ZHANG H, NING A, PRABHAKAR R B, et al. LLMCompass: Enabling efficient hardware design for large language model inference[C]//Proceedings of the 51st Annual International Symposium on Computer Architecture. Buenos Aires: IEEE, 2024: 1080-1096.

[43] LIU L, WANG Y, LI X, et al. PAM: Processing across memory hierarchy for efficient KV-centric LLM serving system[EB/OL]. 2026[2026-09-03]. arXiv:2602.11521.

[44] LI Z, BIAN Z, HUANG Z, et al. TokenStack: A heterogeneous HBM-PIM architecture and runtime for efficient LLM inference[EB/OL]. 2026[2026-09-03]. arXiv:2605.05639.

[45] UCIe CONSORTIUM. UCIe 3.0 specification: Driving innovation for efficient, scalable, and reliable Chiplet integration[EB/OL]. 2025[2026-09-03].

[46] YANG P, ZHANG R, LIU F. Towards efficient large language model serving: A survey on system-aware KV cache optimization[EB/OL]. 2026[2026-09-03]. arXiv:2607.08057.