AI AI Toolkit
Taking Off TodayPythonMIT

Stop Guessing: This Skill Turns Local LLM Tuning Into a Repeatable Science

⭐ 1.7K Stars

What It Actually Does

Ever had this experience: you have a decent GPU sitting in your machine, but your local LLM crawls, so you start hunting through flags, copying parameters from strangers, changing one line and re-running, and in the end you have no real idea whether you actually got faster — and the whole ritual starts over the moment you switch models. inference-optimisation-skill turns that guesswork into a reproducible engineering process. It's an operation manual written for AI agents to read, dedicated to a single question: how do you make a local LLM server faster? It covers single-user and concurrent decoding, speculative decoding, prefill, KV cache sizing, startup time, and time to first token. Crucially, it isn't a list of flags. It distills the author's real tuning runs — including the dead ends — into a fixed set of steps, and it insists that every result stay within a memory cap and survive a full-context stability check, so you don't accidentally tune yourself into a model that falls apart the second a long prompt shows up. The whole point is that the speedups are earned under constraints, not chased with a stopwatch and a prayer, which is exactly what makes them trustworthy when you try to repeat them on your own box.

Why It Took Off

It only went open source on November 16 and racked up more than 900 stars in a matter of days, and I think the reasons are pretty concrete. First, timing: more people are running models locally every month, and pushing 30B-class or even larger models onto consumer cards like Intel Arc or an RTX 3090 is now routine — but most people are stuck at "it runs, but it's slow," and a tuning playbook you can copy wholesale feels like a lifesaver. Second, it lands squarely in the new Agent Skill wave: this isn't documentation aimed at humans, it's a skill that coding agents like Claude Code load, so you hand it the repo and it rolls up its sleeves and starts tuning on its own. The results table in the README is extremely shareable: 2.1× faster 256K prefill, startup cut from 113 seconds to 43, and on 4× H200 a setup that crashed under load reaching 229 tok/s decode. A scoreboard with exact hardware, before-and-after numbers, and model names is far more convincing than any marketing slogan, and it spread through local-LLM communities almost immediately.

The Interesting Part

What's genuinely interesting is the methodology, not any single parameter. The whole thing is six steps. Compute the theoretical ceiling first, so you know where the stop sign is. Change exactly one thing at a time — never bundle edits. Measure every change against interleaved controls to cancel out system noise. Log everything, failures included, because the failures are often the most valuable entries in the whole log. Then gate on long-context retrieval, so a config that looks great on short text but collapses at 128K tokens gets thrown out before anyone trusts it. Finally, before publishing, re-validate on a clean, pinned build. On the technical side it plays with speculative decoding — a small model guesses ahead and a larger one verifies in bulk — multi-token prediction (MTP), custom XPU kernels, KV cache partitioning, and similar hard-won tricks. The table shows the depth of the work: on the same Arc Pro B70, llama.cpp with Q4_K_M manages just 25 tok/s, while vLLM plus custom XPU kernels hits 91.2 tok/s for a single user and 365 at 16 concurrent users. On an RTX 3090, Qwen3.6-35B-A3B at 3 bpw with MTP reaches 251.7 to 357.9 tok/s. Every figure is tagged with its hardware, model, quantization, and engine, so the numbers are reproducible rather than anecdotal — you can argue with the method, but you can't dismiss the measurements.

Who It's For

First, local-model tinkerers — especially anyone holding an Intel Arc card or an RTX 3090 who suspects there's performance still left on the table. Second, teams deploying local inference in production, where what matters is stability and repeatability that survives long contexts, not a one-off benchmark high; a workflow built on controls and failure logs fits that need perfectly. Third, heavy users of Claude Code or other skill-capable coding agents, who can bolt this on as a loadable capability and let the agent do the grinding for them. Even if you never touch an agent at all, the core loop — ceiling, single variable, control group, full log, long-context gate, pinned build — is worth stealing for basically any performance-tuning work you do, whether that's a language model, a database, or a rendering pipeline.

Getting Started

Clone the repo and read SKILL.md, or hand PROMPT.md to any model and let it follow the process step by step. With Claude Code, load it as a skill. The project is a methodology rather than a piece of software, so it doesn't lock you to one engine — vLLM, SGLang, or llama.cpp all work fine as starting points, and the README's reference recipes give you concrete configurations to measure against before you start hill-climbing on your own hardware.

How It Compares

Worth noting: this sits at the methodology layer, not the engine layer, so the fair comparisons are manual tuning and benchmark utilities.

  • Versus manually flipping flags: ad hoc, unrepeatable, failures forgotten. Here the single-variable-plus-control approach turns the process into an experiment anyone can rerun, and the misses get logged too.
  • Versus llama-bench-style tools: a benchmark tells you whether you're fast, not how to get fast. This gives the whole path, from theoretical ceiling all the way to a pinned, validated build.
  • Versus official vLLM/SGLang tuning docs: those cover one engine each; this spans engines and hardware, even measuring the gap between vLLM and SGLang on the very same card and day.
  • Versus built-in auto-tuning: auto-tuners usually search inside a single engine, while this takes in the entire serving stack — quantization, kernels, KV cache, speculative decoding, and startup.

In short, others hand you a fish; this teaches you how to find the school on an unfamiliar card, and to prove you found it.

🚀

Get Started

Open Source · Commercial Friendly

MIT· Python