Xiaohongshu, with Peking University and SJTU, Proposes First Position-Independent Caching System for Hybrid-Attention LLMs

Core Highlights

Xiaohongshu's technical team, together with Peking University and Shanghai Jiao Tong University, has proposed a system called HYPIC, described as the world's first position-independent caching scheme for hybrid-attention large language models. On hybrid-attention architectures, HYPIC reduces first-token latency by an average of 3.25 times. Across four production-grade models, it improves sustainable queries per second (QPS) by 1.66 times under the same service-level objective (SLO), while task quality differs from full recomputation by only 1.71 points. In other words, it achieves a rare balance between engineering gains and output fidelity.

The significance is not merely a benchmark number. Latency and throughput are the two variables that most directly determine whether a large model can be operated as a real product rather than a demo. Cutting first-token latency more than threefold while simultaneously raising throughput is the kind of result that, if reproducible at scale, changes the economics of serving. For a content platform whose business depends on fast, conversational interactions, that translates directly into lower compute bills and snappier user experiences.

Specific Capabilities and What Happened

Conventional KV caching is tightly bound to the absolute position of each token: the moment a prefix changes, the cached state becomes invalid and must be recomputed from scratch, wasting substantial compute. HYPIC's core idea is to decouple 'position' from 'content' so that cached state no longer depends on absolute coordinates, enabling far more flexible reuse. The team validated the approach on four real production models spanning different traffic patterns, demonstrating that the dual benefits of throughput and latency hold under authentic load rather than only in synthetic tests.

This matters because production traffic is messy. Users send partial edits, follow-up questions, and reused context fragments that do not align neatly with fixed prefixes. A position-coupled cache forces expensive recomputation on exactly those realistic patterns. By treating cached entries as position-independent, HYPIC allows the serving system to recognize that the same underlying content, even when placed at different offsets, carries the same semantic state, and to reuse it accordingly.

Technical Details

Hybrid attention refers to models that alternate between full attention and efficient variants such as sliding-window attention. This creates a new caching challenge: states from different attention types are difficult to reuse under a unified scheme. HYPIC's position-independent design lets cached states produced from different prefixes be shared safely across requests. The 1.71-point quality gap means that, compared with token-by-token full recomputation, users can scarcely perceive any loss in accuracy, while gaining a first-token speedup of more than three times.

Technically, the contribution lies in how the system represents and indexes cached keys and values. Rather than keying cache entries by absolute sequence index, HYPIC abstracts the positional dependence out of the storage format, so a cached block can be relocated or recomposed without invalidation. The hybrid-attention complication is handled by separating the caching policy per attention type while preserving a common, position-free addressing layer. The result is a drop-in-friendly mechanism: existing models keep their weights, and only the serving runtime needs to change.

Comparison with Competing Approaches

Today's mainstream inference optimizations revolve around continuous batching, PagedAttention, speculative decoding, and similar techniques, yet cache reuse is still broadly limited by absolute position. HYPIC's differentiation is that it attacks the root cause—position coupling—rather than making scheduling tweaks at the surface. Compared with purely algorithmic approaches such as distillation or quantization, it targets the deployment pain point at the system level and is friendly to already-shipped production models, delivering benefits without retraining.

Relative to prior caching work, HYPIC is notable for being validated on hybrid-attention models specifically, a class that many general-purpose caching methods handle poorly. Where prefix caching and semantic caching reduce recomputation by matching identical sequences, HYPIC goes further by tolerating positional shifts. That positions it closer to an infrastructure primitive than to a model-specific trick, which is why its authors frame it as a general serving-system advance rather than a single-model win.

Industry Impact and Applicable Scenarios

Simply put, HYPIC tackles the chronic 'expensive and slow' disease of deploying large models. For platforms like Xiaohongshu with high concurrency, many models, and rich content formats, first-token latency and sustainable QPS directly determine user experience and compute cost. The work was completed by a company jointly with universities, reflecting the pragmatic industry-academia collaboration path common in China's AI ecosystem.

If position-independent caching is adopted by more inference frameworks, it could become general infrastructure for reducing cost and increasing efficiency in large-model services, while clearing an engineering obstacle to the wider adoption of hybrid-attention architectures. The broader implication is that serving-system research—once the quiet backstage of AI—is becoming a first-class frontier, where marginal gains in cache reuse compound into large savings across fleets of GPUs. For Chinese AI labs operating under compute constraints, such system-level ingenuity is not a luxury but a necessity, and HYPIC is a representative example of how domestic teams are turning constraints into architectural innovation.