Context Adaptive Memory-Efficient LLM Inference for Edge Multi-Agent Systems

Large Language Models (LLMs) excel at multi-document QA, summarization, code generation, and other language-intensive tasks, yet they demand substantial memory resources for storing keyvalue (KV) caches and processing attention in long-context scenarios. These requirements often prohibit on-device or edge deployments in multi-agent systems (MAS), where multiple agents share or update contextual information and need efficient inference pipelines. We present CASK (Context-Adaptive Sparse Key-value), an inference-time strategy that reduces memory usage while preserving strong performance on extended contexts. CASK addresses this challenge with two complementary mechanisms: a dynamic sparse attention module-a lightweight, meta-learned component-that identifies the most relevant context tokens, and an adaptive KV-cache compression technique that dynamically quantizes and prunes less critical key-value pairs based on usage frequency and recency. These innovations enable near-lossless performance on long-context tasks while cutting memory usage by up to 40% and boosting inference speed by as much as 20%. Evaluations on LongBench [2] and multi-agent benchmarks show that CASK maintains over 95% of baseline accuracy while allowing more agents or extended histories under tight GPU budgets. Integration into a vision-language agent for collaborative, multimodal contexts underscores its practicality for resource-constrained LLM deployments in MAS.