Abstract
Agent workflows extend model calls into planning, tools, observations, verification and retries, turning KV into cross-turn execution state. AgentKV uses event graphs to record types and versions, estimates value from dependencies, reuse and recomputation, and chooses HBM, DDR/CXL or NVMe placement, precision and recovery timing. Tool-call gaps enable migration, reversible compression and predictive prefetching. In a discrete-event simulation of 3,000 batches with 32 blocks each and HBM equal to 35% of the working set, AgentKV lowers P95 recovery overhead from 5.59 to 3.77 ms and transfers from 563.10 to 159.49 MB/batch versus LRU, reductions of 32.51% and 71.68%, while hiding 50.28% of recovery cost. These simulated results favor joint lifecycle, reversibility, tail and correctness management over hit rate alone.
LLMs increasingly plan, call tools, observe feedback and revise actions. ReAct interleaves reasoning and actions[2], while Toolformer demonstrates learning tool-use patterns[3]. This changes timing as well as context length: millisecond decoding alternates with second-scale tools, retries and branching checks. KV may be reused seconds later or become unnecessary after a commit. PagedAttention reduces device fragmentation[4] and SGLang reduces repeated prefill through structured programs and prefixes[5], but neither alone decides when agent state should be retained, compressed, moved, verified or retired. Tokencake, Continuum, MORI, PBKV and CommitKV examine tool gaps, TTL, relative idleness, future calls and commits[11][12][13][14][15][16]. AgentKV builds state graphs from control events without reading hidden reasoning text, combining future dependencies, recomputation, precision risk and tool gaps. Contributions include an event-based state model, risk-constrained reversible placement, gap-aware migration/prefetching and reproducible simulation of capacity, tails, transfers and prediction error.
1. Agent KV state and problem definition
1.1 From request cache to execution state
Autoregressive inference repeatedly reads historical keys and values. For batch B, layers L, context T, KV heads Hₖᵥ, head dimension dₕ and bit width b, capacity excluding paging and metadata is:
Equation (1) grows approximately linearly with context and active sessions. A model-call completion does not end agent-state usefulness: the next call often reuses most history. Immediate release requires prefill again, while permanent residency crowds out other sessions. Request completion and state retirement must be separate.
Figure 1. Agent workflow and KV lifecycle
Planning creates actions; tools freeze model execution; observations append tokens and resume decoding; verification and commits change dependencies. Unlike user-paced chat, tool durations are heavy-tailed and alternate between short calls and long blocking periods. MORI describes relative idleness[13], Continuum balances reuse and queuing through TTL[12], and Tokencake coordinates isolation with gap-time uploads[11]. AgentKV works at event-page granularity so prefixes, evidence, failed branches and verified results within one session can receive different policies.
1.2 State classes and lifecycle semantics
A manageable page is sᵢ=〈vᵢ, τᵢ, gᵢ, qᵢ, zᵢ〉: data/positions, semantic type, event node/version, precision/recovery replica, and placement/access/migration state. Types come from prompt boundaries, tool protocols, message roles and orchestration, not classification of hidden chain-of-thought text.
Table 1. Agent state classes and default policies
| State class | Typical source | Lifecycle criterion | Default policy |
|---|---|---|---|
| Fixed prefix | System prompts and tool schemas | Session or tenant version change | Cross-turn sharing at high or stable low precision |
| Task input | User goals and constraints | Task completion or goal revision | Resident while active; demote after branch switches |
| Evidence | Retrieved documents and code | All dependent nodes committed | Expensive to recompute; prioritize reversible replicas |
| Tool observations | Search, database and compiler results | Replaced or committed observation | High short-term value; prefetch during tool gaps |
| Verified results | Passed tests and confirmed calculations | Downstream completion | High correctness weight; avoid irreversible deletion |
| Temporary reasoning | Candidate plans and drafts | Branch commit or timeout | Mixed-precision compression with recomputation entry points |
| Failed branches | Invalid actions and paths | Confirmed failure | Low-priority retirement; retain audit summaries |
Classes can change: observations become cold when superseded, drafts become verified after tests, and old prefixes may serve in-flight work after an upgrade but not new tasks. Active, frozen, compressed, restoring and committed/retired states are system actions whose legality follows the event graph.
Figure 2. AgentKV transitions and control conditions
1.3 Limits of existing policies
LRU cannot distinguish imminent short-tool returns from long blocks. LFU/ARC adapt to frequencies[30], but infrequently accessed critical state may remain valuable. TTL lacks intra-session page distinctions. H2O, SnapKV and StreamingLLM use attention signals primarily for single-sequence decoding quality[21][22][23]. IntentKV adds cross-turn intent memory[17], CommitKV identifies completed state through commit-related deletion effects[16], and VeriCache verifies against full KV for output equivalence[18]. These supply useful components but not a unified interface for dependencies, tiers, reversibility, tool windows and SLOs.
Table 2. KV management design dimensions
| Method class | Main signal | Granularity | Reversibility | Tool gaps | Task dependencies |
|---|---|---|---|---|---|
| Paging / prefix caching | Address maps and identical prefixes | Block / request | Not applicable | No | Weak |
| Attention eviction | Attention scores and heavy hitters | Token / layer | Usually irreversible | No | None |
| TTL / idleness | Expected return and program activity | Session | Offload and restore | Yes | Weak |
| Call prediction | Future agents or tool sequences | Workflow / entry | Recoverable | Partial | Medium |
| Commit-aware compression | Deletion effects around commits | Event page | Usually deletion | No | Strong |
| AgentKV | Dependencies, reuse, recomputation, risk and idle windows | Event page | Compressed replicas plus verification / recomputation | Yes | Strong |
2. AgentKV design
2.1 Architecture
AgentKV sits between orchestration and inference without changing model parameters or attention semantics. The orchestrator sends structured task, model, tool, observation, branch, verification, rollback and completion events. The engine exposes page handles, physical maps, compression and asynchronous DMA. Five modules cover event annotation, lifecycle prediction, SLO/risk control, tiered reversible placement and gap-aware migration/prefetching.
Figure 3. AgentKV architecture
The event graph is a DAG or bounded-back-edge graph of model, tool, observation and verification nodes with data/control edges. Pages track producers and candidate consumers. Retirement requires all reachable consumers to finish and audit/rollback retention to end. Loops use versioned windows and conservative summary nodes to bound graph growth.
2.2 Lifecycle prediction and value
Features include state type, time since events, task depth, successors, tool category/duration quantiles, branch confidence, prefix sharing, recomputation tokens and queue pressure. Predictions include next-access distribution Fᵢ(t), reuse probability and confidence. Calibrated boosted trees or small MLPs can limit critical-path overhead; low confidence triggers conservative fallback.
State value balances expected savings against the opportunity cost of fast-memory occupancy:
Cᵢʳᵉᶜᵒᵐᵖ is prefill/tool recomputation cost; Dᵢ measures unfinished dependencies and critical-path weight; Qᵢ captures correctness impact; Cᵢᵐᵒᵛᵉ includes DMA, compression and synchronization; Cᵢᵒᶜᶜᵘᵖʸ represents HBM-induced queuing. SLOs and task classes set coefficients: code repair emphasizes correctness/recomputation, while open retrieval may compress temporary evidence more aggressively.
2.3 Tiered placement and reversible compression
Pages may reside in HBM, DDR/CXL or NVMe as high precision, INT8, INT4, low-rank state or recomputation entry points. KIVI and KVQuant demonstrate capacity savings[19][20], and CacheGen combines compression with network streaming[24]. Agent errors can accumulate into invalid code, JSON or tool arguments. AgentKV separates working and reference copies: low-risk pages may retain only compressed state, while high-risk pages keep full lower-tier replicas for checkpoint verification or recomputation.
kᵢ and qᵢ denote tier and precision; Lᵢ recovery latency, Eᵢ movement/compression energy, Rᵢ risk bound and Cₖ capacity. Online integer optimization is replaced by two stages: rank HBM candidates by Vᵢ/mᵢ, then select evicted pages' precision and destination using Lagrangian costs. Only event-affected candidates are recomputed, approximately O(n log n).
2.4 Tool-gap migration and predictive prefetching
Tool calls provide overlap windows. Rather than immediately offload a session, the system ranks pages by expected idleness and dependencies. For total migration/compression/restoration time Tᵢᵐᵒᵛᵉ and gap Tᵢᵍᵃᵖ, the hidden fraction is:
Short gaps move only small, low-value pages; longer waits progressively admit more. Prefetching works backward from predicted returns and reserves credits. Early returns can cancel unfinished prefetches while execution uses the minimum ready prefix. MORI partitions GPU/CPU queues by relative idleness[13] and PBKV predicts future agent calls[14]. AgentKV predicts event pages and constrains premature demotion of valuable state through risk control.
Figure 4. Tool idle time and hidden migration fraction
2.5 Consistency, versions and security
Page identity combines model, adapter, tokenizer, RoPE, tenant, session, position and event versions. Zero-copy reuse requires complete compatibility. Updated schemas/prompts leave old pages read-only for in-flight work before reclamation. Remote pages require encryption, access control and confirmed erasure; directories must not expose other tenants' prompt lengths, tools or access times. TokenDance demonstrates shared blocks and differential storage[15], but stronger sharing requires explicit consistency and side-channel controls.
3. Discrete-event simulation
3.1 Purpose and scope
The simulation tests trends without vendor-specific hardware or full production traces. It does not claim real-cluster throughput. It asks whether dependencies improve placement under identical budgets, compression/overlap reduce visible recovery and transfers, and prediction errors cause instability. Seeds, distributions, bandwidth and formulas are fixed for reproduction.
Table 3. Simulation parameters
| Parameter | Setting | Notes |
|---|---|---|
| Simulation batches | 3,000 batches | An independent concurrent working set per batch |
| Blocks per batch | 32 | Sampled from seven event-state categories |
| State size | 8~220 MB | Truncated lognormal distribution |
| Tool idle time | Heavy-tailed lognormal | Longer means for evidence and observations |
| Host restoration bandwidth | 24 GB/s | Includes 1.2 ms fixed overhead and queue noise |
| AgentKV compression ratio | 3.2∶1 | Includes compression, decompression and restoration |
| HBM budget | 20%–70% of working set | Main results use 35% |
| Prediction noise | Log error 0.05–0.90 | Main standard deviation: 0.35 |
State proportions are system prefixes 7%, tool definitions 10%, evidence 18%, observations 20%, verified results 12%, drafts 23% and failed branches 10%. Types have distinct reuse, recomputation and correctness weights: fixed-prefix reuse is 0.99 versus 0.12 for failures; verified results carry the highest quality/recomputation weights. This deliberately separates temporal locality from value and does not fit a specific product.
3.2 Policies and metrics
LRU ranks recency; TTL uses expected idleness and size; semantic priority uses type, approximate importance and correctness; AgentKV uses Equation (3), 3.2× compression and gap prefetching. The first three restore full-precision host pages; AgentKV restores compressed lower-tier copies with partial overlap.
Metrics include byte hit rate, P50/P95/P99 recovery overhead, transfers per batch, hidden fraction and a risk proxy. The proxy combines relative prediction error, correctness weight and demotion severity; it indicates pressure for verification/high-precision fallback, not task accuracy or a substitute for application tests.
4. Results and analysis
4.1 HBM-capacity sensitivity
Figure 5. Byte hit rate by HBM budget
Hit rates rise as budgets grow from 20% to 70%. Semantic priority usually has slightly higher hits; AgentKV is slightly lower at 20%–40% but has lower visible recovery cost. Compressed misses restored during tool waits can be cheaper than extra hits without prefetching, so hit rate alone is insufficient.
Figure 6. Recovery P95 by HBM budget
AgentKV has lower P95 throughout: 4.07 versus LRU's 5.80 ms at 20%, and 3.01 versus 4.75 ms at 70%. More capacity helps LRU through hits, while compression and gaps continue helping AgentKV. Large or critical remaining misses explain why tails do not decline in direct proportion to hit rates.
4.2 Main configuration
Table 4. Simulation at 35% HBM
| Policy | P50/ms | P95/ms | P99/ms | Byte hit rate / % | Transfers / MB per batch | Hidden fraction / % |
|---|---|---|---|---|---|---|
| LRU | 2.66 | 5.59 | 7.45 | 34.70 | 563.10 | 0.00 |
| TTL | 2.61 | 5.75 | 7.78 | 33.71 | 571.69 | 0.00 |
| Semantic priority | 0.00 | 5.83 | 7.84 | 42.99 | 491.68 | 0.00 |
| AgentKV | 0.00 | 3.77 | 4.68 | 40.82 | 159.49 | 50.28 |
AgentKV lowers P95 from 5.59 to 3.77 ms versus LRU, or 32.51%, and P99 by 37.16%. Transfers fall from 563.10 to 159.49 MB/batch, or 71.68%. Its 40.82% byte hit rate is below semantic priority's 42.99%, yet that policy has P95 5.83 ms. Optimize recovery cost times visible fraction, not physical hits alone.
P50 is zero because over half the reused state is resident or fully restored during gaps. The 50.28% hidden fraction averages all offloaded, later-reused pages; it does not imply every tool hides half its transfer. Short calls, late predictions and bandwidth competition still create tails. Real scheduling must include PCIe/CXL competition with decoding and networking.
4.3 Prediction error and risk
Figure 7. Prediction error and AgentKV tails
As log-error deviation grows from 0.05 to 0.90, P95 remains 3.70–3.79 ms. Value also uses graph dependencies, recomputation and size, and low-confidence pages retain replicas or recomputation paths rather than irreversible deletion. The risk proxy rises from 1.05% to 12.16%, exposing increased need for conservative placement and verification rather than concealing correctness risk behind stable latency.
4.4 Ablation analysis
Ablations use full AgentKV as the normalized baseline. Without gap prefetching, bytes stay constant but recovery becomes visible. Without reversible compression, transfers return to full-KV levels. Without dependencies, recently accessed but committed state may remain. Without risk control, lower average latency loses error bounds for code and tools. Table 5 gives directional component contributions from the same model.
Table 5. Normalized AgentKV ablation
| Configuration | P95 latency | Transfer volume | High-value mistaken-eviction proxy | Notes |
|---|---|---|---|---|
| Full AgentKV | 1.00 | 1.00 | 1.00 | Events, compression, prefetching and risk control enabled |
| Without tool-gap prefetching | 1.41 | 1.00 | 1.00 | Exposed recovery increases tails |
| Without reversible compression | 1.27 | 3.20 | 0.92 | Full KV transfers preserve precision but consume bandwidth |
| Without event dependencies | 1.18 | 1.09 | 1.64 | Reuse/time ranking loses commit semantics |
| Without risk control | 0.94 | 0.91 | 2.73 | Aggressive demotion unsuitable for correctness-sensitive tasks |
| Fixed precision | 1.12 | 1.36 | 1.08 | Cannot adapt representation to state criticality |
5. Prototype and engineering considerations
5.1 Inference-engine interfaces
A minimal implementation can extend PagedAttention blocks: attach event IDs to messages/tools, record IDs, positions and compression descriptors in page tables, migrate asynchronously and restore logical mappings. SGLang event nodes can reference radix-tree ranges to avoid duplicating prefixes. LMCache and Mooncake demonstrate external KV tiers[8][10]; AgentKV adds event semantics, risk descriptors and scheduling objectives.
Recompute value at tool, observation, commit or pressure events rather than per token. Compression can use idle GPU streams or host vectors. CXL holds shared warm state, but simultaneous restoration needs bandwidth control. Session-level HBM, host and link credits can queue excess workflows or select lower precision.
5.2 Correctness and observability
Small numerical errors may become code, SQL, JSON or control-flow failures over long decoding. VeriCache verifies draft tokens against full KV[18]. AgentKV can apply this to high-risk pages, with compressed drafting and full-copy checkpoint checks or recomputation. Monitor precision, recovery source, rejection rates, recomputed tokens, task success and format errors, not memory savings alone.
Labels use observable protocol boundaries—roles, hashed tool names, call IDs, status codes, tests and graph edges—not hidden reasoning. Evidence importance can use dependencies and references instead of storing/classifying sensitive text. Long-term auditing may retain summaries and hashes while deleting KV under retention rules.
5.3 Applicability and limitations
Table 6. AgentKV applicability and limits
| Aspect | Applicable conditions | Main limitations | Improvement direction |
|---|---|---|---|
| Workflow | Explicit tool/event boundaries | Free conversation lacks task graphs | Learn weak dependencies from request logs |
| Storage | Host memory, CXL or NVMe tiers | Link contention may offset benefits | Bandwidth credits and congestion feedback |
| Compression | KV quantization or low-rank support | Long-output correctness is difficult to guarantee statically | Full-copy verification and online fallback |
| Prediction | Locally stable tool durations and call structure | Failures and new-tool distribution shifts | Confidence calibration and conservative fallback |
| Sharing | Compatible model, positions and prefix identity | Tenant leakage and version contamination | Encryption, isolation and generational directories |
| Evaluation | Available task-success and tail metrics | Simulation cannot replace real clusters | Public traces, prototypes and end-to-end benchmarks |
Quantitative results use synthetic events and analytical transfers, not measured GPU/model/framework performance. Real outcomes depend on layouts, PCIe/CXL, NUMA, parallelism, tree fragmentation, tool concurrency and kernels. Future work should publish event/duration traces, prototype in vLLM or SGLang, and use tasks such as SWE-bench and OSWorld[33][34] to jointly measure completion time, success and output equivalence.
6. Conclusion
Agents turn KV into state spanning turns, tools and branches. AgentKV combines event dependencies, versions, reuse, recomputation, correctness and movement costs to select tier and precision. Reversible compression reduces transfers, tool gaps hide recovery, and risk triggers full-copy checks or recomputation. At 35% HBM in simulation, it reduces P95 overhead 32.51% and transfers 71.68% versus LRU. The central conclusion is auditable joint management of tails, lifecycle, reversibility and correctness rather than hit rate alone. Real traces, network congestion, model-level verification and tenant isolation remain prototype priorities.
Lifecycle-aware KV cache for agentic inference
Abstract Agentic inference expands a single model invocation into a long-lived workflow of planning, tool execution, observation, verification, and retry. Consequently, the KV cache becomes cross-turn execution state rather than a request-local tensor. This paper presents AgentKV, a lifecycle-aware KV-cache system that annotates state pages with event-graph dependencies and versions, predicts reuse and recomputation costs, and jointly selects storage tier, compression precision, and restoration time across GPU HBM, host DDR/CXL memory, and NVMe. AgentKV further exploits tool-call gaps for migration, reversible compression, and predictive prefetching, while a risk controller preserves full references or recomputation checkpoints for correctness-sensitive states. In an event-level discrete simulation with 3,000 batches and 32 state blocks per batch, under an HBM budget equal to 35% of the working set, AgentKV reduces the P95 visible restoration delay from 5.59 ms to 3.77 ms and cross-tier transfer from 563.10 MB to 159.49 MB per batch compared with LRU, while hiding 50.28% of restoration work. The results show that agentic cache management should optimize lifecycle, reversibility, tail latency, and correctness risk jointly rather than maximizing hit rate alone.
References
[1] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems. Long Beach: Curran Associates, 2017, 30: 5998-6008.
[2] YAO S, ZHAO J, YU D, et al. ReAct: synergizing reasoning and acting in language models[C]//International Conference on Learning Representations. Kigali: OpenReview, 2023.
[3] SCHICK T, DWIVEDI-YU J, DESSI R, et al. Toolformer: language models can teach themselves to use tools[C]//Advances in Neural Information Processing Systems. New Orleans: Curran Associates, 2023, 36: 68539-68551.
[4] KWON W, LI Z, ZHUANG S, et al. Efficient memory management for large language model serving with PagedAttention[C]//Proceedings of the 29th Symposium on Operating Systems Principles. New York: ACM, 2023: 611-626.
[5] ZHENG L, YIN L, XIE Z, et al. SGLang: efficient execution of structured language model programs[C]//Advances in Neural Information Processing Systems. Vancouver: Curran Associates, 2024, 37.
[6] AGRAWAL A, KARSAVURAN M O, KOLLAR T, et al. Taming throughputs-latency tradeoff in LLM inference with Sarathi-Serve[C]//18th USENIX Symposium on Operating Systems Design and Implementation. Santa Clara: USENIX Association, 2024: 117-134.
[7] ZHONG Y, LIU S, CHEN J, et al. DistServe: disaggregating prefill and decoding for goodput-optimized large language model serving[C]//18th USENIX Symposium on Operating Systems Design and Implementation. Santa Clara: USENIX Association, 2024: 193-210.
[8] QIN R, LI Z, HE W, et al. Mooncake: a KVCache-centric disaggregated architecture for LLM serving[C]//23rd USENIX Conference on File and Storage Technologies. Santa Clara: USENIX Association, 2025.
[9] PAN X, QIU M, TANG W, et al. KVFlow: efficient prefix caching for accelerating LLM-based multi-agent workflows[EB/OL]. arXiv:2507.07400, 2025[2026-09-03]. https://arxiv.org/abs/2507.07400.
[10] CHENG C, WANG Z, LIU Y, et al. LMCache: an efficient KV cache layer for enterprise-scale LLM inference[EB/OL]. arXiv:2510.09665, 2025[2026-09-03]. https://arxiv.org/abs/2510.09665.
[11] BIAN Z, WU F, MA T, et al. Tokencake: a KV-cache-centric serving framework for LLM-based multi-agent applications[EB/OL]. arXiv:2510.18586, 2025[2026-09-03]. https://arxiv.org/abs/2510.18586.
[12] LI H, HE R, MANG Q, et al. Continuum: efficient and robust multi-turn LLM agent scheduling with KV cache time-to-live[EB/OL]. arXiv:2511.02230, 2025[2026-09-03]. https://arxiv.org/abs/2511.02230.
[13] XIA T, LI H, LI Z, et al. Idleness is relative: exploiting tool-call idle windows for offloading in agentic systems with MORI[EB/OL]. arXiv:2606.00866, 2026[2026-09-03]. https://arxiv.org/abs/2606.00866.
[14] ZHENG H, FU F, WU J, et al. Efficient serving for dynamic agent workflows with prediction-based KV-cache management[EB/OL]. arXiv:2605.06472, 2026[2026-09-03]. https://arxiv.org/abs/2605.06472.
[15] BIAN Z, WU F, ZHANG C, et al. TokenDance: scaling multi-agent LLM serving via collective KV cache sharing[EB/OL]. arXiv:2604.03143, 2026[2026-09-03]. https://arxiv.org/abs/2604.03143.
[16] HUANG W, ZHANG J, ZHENG X. CommitKV: lifecycle-aware KV cache compression via commit transitions for multi-turn agents[EB/OL]. arXiv:2608.07855, 2026[2026-09-03]. https://arxiv.org/abs/2608.07855.
[17] LI J, LOU J, LI J. IntentKV: cross-turn intent-aware KV cache pruning for agent inference[EB/OL]. arXiv:2606.09916, 2026[2026-09-03]. https://arxiv.org/abs/2606.09916.
[18] YAO J, SHEN S, DU K, et al. VeriCache: turning lossy KV cache into lossless LLM inference[EB/OL]. arXiv:2605.17613, 2026[2026-09-03]. https://arxiv.org/abs/2605.17613.
[19] LIU Z, YUAN J, JIN H, et al. KIVI: a tuning-free asymmetric 2bit quantization for KV cache[C]//International Conference on Machine Learning. Vienna: PMLR, 2024.
[20] HOOPER C, KIM S, MOHAMMADZADEH H, et al. KVQuant: towards 10 million context length LLM inference with KV cache quantization[C]//Advances in Neural Information Processing Systems. Vancouver: Curran Associates, 2024, 37.
[21] ZHANG Z, SHENG Y, ZHOU T, et al. H2O: heavy-hitter oracle for efficient generative inference of large language models[C]//Advances in Neural Information Processing Systems. New Orleans: Curran Associates, 2023, 36: 34661-34710.
[22] XIAO G, TIAN Y, CHEN B, et al. Efficient streaming language models with attention sinks[C]//International Conference on Learning Representations. Vienna: OpenReview, 2024.
[23] LI Y, HUANG Y, YANG B, et al. SnapKV: LLM knows what you are looking for before generation[C]//Advances in Neural Information Processing Systems. Vancouver: Curran Associates, 2024, 37.
[24] LIU Y, LI H, CHENG Y, et al. CacheGen: KV cache compression and streaming for fast large language model serving[C]//Proceedings of the ACM SIGCOMM 2024 Conference. New York: ACM, 2024: 38-56.
[25] LEE W, LEE J, KIM J, et al. InfiniGen: efficient generative inference of large language models with dynamic KV cache management[C]//18th USENIX Symposium on Operating Systems Design and Implementation. Santa Clara: USENIX Association, 2024: 155-172.
[26] JIANG H, WU Q, LIN C Y, et al. LLMLingua: compressing prompts for accelerated inference of large language models[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: ACL, 2023: 13358-13376.
[27] DING Y, JIANG L, ZHENG X, et al. LongLLMLingua: accelerating and enhancing LLMs in long context scenarios via prompt compression[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Bangkok: ACL, 2024: 1658-1677.
[28] CHEN L, ZAHARIA M, ZOU J. FrugalGPT: how to use large language models while reducing cost and improving performance[EB/OL]. arXiv:2305.05176, 2023[2026-09-03]. https://arxiv.org/abs/2305.05176.
[29] SHENG Y, ZHENG L, YUAN B, et al. FlexGen: high-throughput generative inference of large language models with a single GPU[C]//International Conference on Machine Learning. Honolulu: PMLR, 2023, 202: 31094-31116.
[30] MEGIDDO N, MODHA D S. ARC: a self-tuning, low overhead replacement cache[C]//2nd USENIX Conference on File and Storage Technologies. San Francisco: USENIX Association, 2003: 115-130.
[31] COMPUTE EXPRESS LINK CONSORTIUM. Compute Express Link specification revision 3.1[S/OL]. Beaverton: CXL Consortium, 2023[2026-09-03].
[32] OPENAI. A practical guide to building agents[R/OL]. San Francisco: OpenAI, 2025[2026-09-03].
[33] JIMENEZ C E, YANG J, WETTIG A, et al. SWE-bench: can language models resolve real-world GitHub issues?[C]//International Conference on Learning Representations. Vienna: OpenReview, 2024.
[34] XIE T, ZHANG D, CHEN J, et al. OSWorld: benchmarking multimodal agents for open-ended tasks in real computer environments[C]//Advances in Neural Information Processing Systems. Vancouver: Curran Associates, 2024, 37.
