AI AI Toolkit
China AI tip

ModelBest's LES lets a 35B-A3B MoE run lossless on-device at 20+ tok/s on iPhone

📰 公众号:面壁智能(MiniCPM) 📅 2026-10-20

Core Highlights

The ModelBest MiniCPM-V team introduced Lookahead Expert Scheduling (LES), the technique behind MiniCPM-V 4.7's on-device execution, claiming the first real-time lossless run of a 35B-A3B-scale MoE on the edge, measured at over 20 tokens/s decode on an iPhone 18 Pro Max. This is a hard-metric-level breakthrough for "putting large models on phones," meaning the edge is no longer just for small models but can run genuinely weighty ones without cloud help.

What Happened

So-called "on-device real-time lossless" means, without shrinking the model or falling back to the cloud, a 35B-total / 3B-active MoE truly runs at usable speed locally on a phone. LES's "lookahead" shows in this: it doesn't schedule only after a token appears, but predicts which experts to activate next in advance, reducing wait and turning inference into a continuous, smooth process. This removes the jarring pauses that usually make on-device models feel broken to end users.

Technical Details

MoE (Mixture-of-Experts) models split parameters into multiple "experts," activating only part each time. The difficulty is the overhead of expert routing and loading. LES uses lookahead scheduling to precompute "which experts next" and pre-move them into cache, pushing decode latency above 20 tokens/s. The iPhone 18 Pro Max's NPU and unified memory bandwidth are the hardware premise that lets it run, also showing edge deployment depends heavily on chip capability and that silicon and software must co-design.

Compared With Alternatives

Many edge solutions trade speed via extremely low-bit quantization or heavy pruning, at the cost of noticeable accuracy loss (lossy). LES emphasizes "lossless" — speeding up without sacrificing quality — taking a different technical route from Apple Intelligence and Qualcomm NPU approaches, valuing experience over mere parameter compression. The claim of lossless is what separates a marketing spec from a usable product.

Industry Impact / Use Cases

For phone makers and app developers, this means offline, privacy-friendly local multimodal assistants are closer to reality. For the industry, on-device large models moving from "can run" to "run smoothly" will directly drive the landing of on-device agents, real-time translation, local image understanding, and other scenarios, letting phones truly possess local intelligence.

Practical Notes

The phrase "lossless" is doing a lot of work and is worth holding to a standard. Most edge claims quietly trade away accuracy to hit a speed number; if LES genuinely holds quality while hitting 20+ tokens/s on a phone, that is the rare case where the spec and the experience align. The mechanism — predicting and pre-loading the next experts — is the kind of co-design between scheduler and silicon that only pays off when you control both the software and the target hardware, which is why this result comes from a team building for a specific device class.

What to Watch

Two things to watch. First, whether 20 tokens/s holds beyond the demo: real apps juggle UI, memory pressure, and thermal throttling, and a benchmark on a flagship phone says little about a mid-range device two years old. Second, whether "lossless" survives the messy inputs of real use — long sessions, mixed languages, and low-light camera input are where edge models usually degrade, and lookahead scheduling's predictions are only as good as the routing heuristic under those conditions.

Closing

LES is a concrete step in the shift from "can we fit a big model on a phone" to "can it run well enough that people forget it's local." If the lossless claim holds up in the wild, it changes the calculus for on-device AI: less cloud dependency, better privacy, and assistants that work on a plane. The broader signal is that the edge is becoming a serious frontier for model efficiency, not just a place to run tiny models.

Why It Matters Now

On-device AI has been stuck in a two-step limbo: models small enough to run were too weak to matter, and models strong enough to matter were too big to run. LES pushes the boundary by proving a 35B-class MoE can decode at usable speed on a phone without sacrificing quality — the rare case where the marketing claim and the user experience align. If it holds, the implication is that phones stop being thin clients for the cloud and start being genuine local intelligence, with the privacy and offline benefits that implies. For an industry chasing bigger data centers, the reminder that silicon-and-software co-design can unlock the edge is a useful counterweight.

The Road Ahead

The caveats are real and worth holding the team to. A benchmark on a flagship iPhone says little about a mid-range device two years old, where thermal throttling and memory pressure quietly erase the gains. And "lossless" must survive messy real inputs — long sessions, mixed languages, low-light camera frames — where edge models usually degrade and lookahead predictions lose accuracy. If those hold up in shipping products, on-device agents, real-time translation, and local vision stop being demos and start being features people rely on. If not, LES remains a bright result on a benchmark rather than a turning point for the edge.