TheTom/turboquant_plus

• highabout#1

  Add a clear repository description

  Why:

  COPY-PASTE FIX

  Copy

  Research and reference implementation for TurboQuant+, a KV cache compression technique for large language models (LLMs) using PolarQuant and Walsh-Hadamard rotation, achieving 3.8-6.4x compression.

  OpenDoneDismiss
• hightopics#2

  Add relevant topics for LLM optimization

  Why:

  COPY-PASTE FIX

  Copy

  llm, kv-cache, transformer, quantization, compression, machine-learning, deep-learning, vllm, llama-cpp, long-context

  OpenDoneDismiss
• highreadme#3

  Move core value proposition to the top of the README

  Why:

  CURRENT

  # TurboQuant+
  
  > 🚀 TurboQuant KV cache compression is now in vLLM (PR #38479, merged April 2026): `--kv-cache-dtype turboquant_k8v4` and friends, with fused Triton store/decode kernels. The PR discussion drew on the asymmetric K/V findings from this repo. **Upstream llama.cpp has merged the core idea too**: Hadamard KV cache rotation (#21038, citing TurboQuant directly) with fast WHT kernels on CPU (#22631), CUDA (#23615), and Vulkan (#23687). Rotation + the stock q4_0 cache is essentially turbo4's rotation stage; the PolarQuant codebook, norm extraction, and asymmetric policies remain here and in the fork.
  
  > ### [Getting Started Guide](docs/getting-started.md) | [Configuration Recommendations](docs/turboquant-recommendations.md) | [Benchmarks](docs/benchmarks.md) | Commercial Support
  
  Implementation of TurboQuant (ICLR 2026) with implementation work, experiments, and follow-on findings beyond the base paper. Compresses transformer KV cache **3.8-6.4x** using PolarQuant + Walsh-Hadamard rotation, at near q8_0 prefill speed and ~0.9x decode throughput at long context. Validated end-to-end from 1.5B to **104B at 128K context on a MacBook** (turbo3, PPL 4.024, 74 GB peak memory).
  
  This repository is the **research home**: the Python reference implementation, the validation papers, and the benchmark data. To run TurboQuant in an inference engine, pick from the table below. Pieces that prove useful and stable get upstreamed incrementally as small, reviewable patches.

  COPY-PASTE FIX

  Copy

  # TurboQuant+
  
  Implementation of TurboQuant (ICLR 2026) with implementation work, experiments, and follow-on findings beyond the base paper. Compresses transformer KV cache **3.8-6.4x** using PolarQuant + Walsh-Hadamard rotation, at near q8_0 prefill speed and ~0.9x decode throughput at long context. Validated end-to-end from 1.5B to **104B at 128K context on a MacBook** (turbo3, PPL 4.024, 74 GB peak memory).
  
  > 🚀 TurboQuant KV cache compression is now in vLLM (PR #38479, merged April 2026): `--kv-cache-dtype turboquant_k8v4` and friends, with fused Triton store/decode kernels. The PR discussion drew on the asymmetric K/V findings from this repo. **Upstream llama.cpp has merged the core idea too**: Hadamard KV cache rotation (#21038, citing TurboQuant directly) with fast WHT kernels on CPU (#22631), CUDA (#23615), and Vulkan (#23687). Rotation + the stock q4_0 cache is essentially turbo4's rotation stage; the PolarQuant codebook, norm extraction, and asymmetric policies remain here and in the fork.
  
  > ### [Getting Started Guide](docs/getting-started.md) | [Configuration Recommendations](docs/turboquant-recommendations.md) | [Benchmarks](docs/benchmarks.md) | Commercial Support
  
  This repository is the **research home**: the Python reference implementation, the validation papers, and the benchmark data. To run TurboQuant in an inference engine, pick from the table below. Pieces that prove useful and stable get upstreamed incrementally as small, reviewable patches.

  OpenDoneDismiss

Category GEO backends resolved for this scan: google/gemini-2.5-flash, deepseek/deepseek-v4-flash

• CATEGORY QUERY

  How to reduce transformer KV cache memory usage for long context LLMs?

  you: not recommended
  AI recommended (in order):

  1. vLLM (vllm-project/vllm)
  2. DeepSpeed-MII (microsoft/DeepSpeed)
  3. Hugging Face Optimum (huggingface/optimum)
  4. Triton Inference Server (triton-inference-server/server)
  5. FasterTransformer (NVIDIA/FasterTransformer)
  6. Hugging Face Transformers (huggingface/transformers)
  7. LongFormer
  8. BigBird
  9. FlashAttention-2 (Dao-AILab/flash-attention)
  10. PyTorch (pytorch/pytorch)
  11. TensorFlow (tensorflow/tensorflow)

  AI recommended 11 alternatives but never named TheTom/turboquant_plus. This is the gap to close.

  Show full AI answer
• CATEGORY QUERY

  Looking for methods to efficiently run large language models on devices with limited RAM.

  you: not recommended
  AI recommended (in order):

  1. llama.cpp (ggerganov/llama.cpp)
  2. AWQ (mit-han-lab/awq)
  3. GPTQ (IST-DASLab/gptq)
  4. bitsandbytes (TimDettmers/bitsandbytes)
  5. Hugging Face Transformers (huggingface/transformers)
  6. accelerate (huggingface/accelerate)
  7. TinyLlama (PKU-YuanGroup/TinyLlama)
  8. LoRA (microsoft/LoRA)
  9. Hugging Face PEFT library (huggingface/peft)
  10. ONNX Runtime (microsoft/onnxruntime)
  11. OpenVINO (openvinotoolkit/openvino)
  12. TensorFlow Lite (tensorflow/tensorflow)
  13. Hugging Face Optimum (huggingface/optimum)

  AI recommended 13 alternatives but never named TheTom/turboquant_plus. This is the gap to close.

  Show full AI answer

• Metadata completeness

  fail

  Suggestion:

• README presence

  pass

• Compared to common alternatives in this category, what is the core differentiator of TheTom/turboquant_plus?

  pass
  ⚠AI did not name TheTom/turboquant_plus — likely talking about a different project

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• If a team adopts TheTom/turboquant_plus in production, what risks or prerequisites should they evaluate first?

  pass
  ✓AI named TheTom/turboquant_plus explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• In one sentence, what problem does the repo TheTom/turboquant_plus solve, and who is the primary audience?

  pass
  ✓AI named TheTom/turboquant_plus explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?