Easy control for Key-Value Constrained Generative LLM Inference(https://arxiv.org/abs/2402.06262)
-
Updated
Feb 13, 2024 - Python
#
cache-eviction
Here are 17 public repositories matching this topic...
A prefix-cache advisor for LLM serving infrastructure that recommends KV-cache capacity and eviction policies from your request traces/logs.
-
Updated
Aug 7, 2026 - Python
A learned, cost-aware eviction policy for semantic LLM caches that replaces heuristic policies like LRU/FIFO with a lightweight offline-trained model, improving regeneration-token savings while maintaining safe LRU fallback.
python machine-learning reinforcement-learning embeddings faiss rag llm semantic-cache llm-inference cache-eviction smart-evictions gptcache smartevict
-
Updated
Jul 28, 2026 - Python
Trace-driven research harness for KV-cache hierarchy policy evaluation in long-context LLM inference.
benchmarking simulation memory-hierarchy performance-modeling kv-cache long-context llm-serving llm-inference cache-eviction systems-research
-
Updated
Apr 14, 2026 - Python
REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression
machine-learning compression retrieval transformers inference budget attention-mechanism reasoning eviction kv-cache attention-heads llm long-context cache-eviction longbench
-
Updated
Apr 15, 2026 - Python
Event-driven simulator for prefix KV-cache eviction policies in LLM serving systems
-
Updated
Jul 8, 2026 - Python
Trace-driven simulator for learned KV Cache eviction on synthetic sparse block-level access traces
-
Updated
Jun 10, 2026 - Python
LAFC-Evict: counterfactual supervision dataset and benchmark for learned cache eviction
caching benchmark machine-learning dataset systems counterfactual cache-eviction learning-augmented-algorithms
-
Updated
Sep 18, 2026 - Python
A small caching library implementing standard caching strategies, starting with LRU, in Python. Part of the multi-language mnemosyne-cache project. Built by hand to develop real, working fluency in caching mechanics rather than just theory.
-
Updated
Sep 5, 2026 - Python
Benchmark of KV-cache eviction policies on GPT-2-medium showing two clear regimes: sink-preserving heuristics dominate at tiny budgets, while attention-based eviction becomes near-lossless at moderate budgets. attention_384 matches full-cache quality with ~33% less cache.
benchmarking transformers pytorch perplexity kv-cache llm-inference attention-sink cache-eviction gpu-profiling
-
Updated
Jul 14, 2026 - Python
Deterministic decay-and-eviction policy for bounded agent memory, and the paired-statistics harness that scores it against LRU, LFU, ARC, a branch-aware baseline and a Belady ceiling.
-
Updated
Aug 29, 2026 - Python
Engineering library for Finite Readout Acceleration (FRA): a quotient framework for deciding when a distinction is safe to stop computing. Companion to Lahtee 2026.
caching performance-engineering web-performance coq quotient-types formal-verification streaming-algorithms cache-eviction
-
Updated
Sep 18, 2026 - Python
FreCoS: freshness- and cost-aware extension to GPTCache (BSc course project)
python machine-learning research-project large-language-models semantic-cache cache-eviction llm-caching gptcache
-
Updated
Aug 9, 2026 - Python
Experimental MSR eviction and semantic prefetch for vLLM KV-cache offloading
-
Updated
Sep 17, 2026 - Python
Real-time AI memory orchestrator for multi-model GPU serving. PERC: provably optimal KV cache eviction via fractional knapsack. 25-79% eviction cost reduction vs LRU.
-
Updated
Apr 22, 2026 - Python
Memory-aware LLM inference optimizer for KV cache compression, eviction, and scheduling.
-
Updated
Apr 13, 2026 - Python
Learning-augmented cache eviction research: decision-aligned eviction-value prediction, evaluated against classical and learned baselines. Companion code for a Knowledge-Based Systems manuscript (second-round revision).
-
Updated
Sep 16, 2026 - Python
Add this topic to your repo
To associate your repository with the cache-eviction topic, visit your repo's landing page and select "manage topics."