spcl/QuaRot

• hightopics#1

  Add specific topics to clarify the project's domain

  Why:

  COPY-PASTE FIX

  Copy

  llm-quantization, 4-bit-inference, large-language-models, deep-learning, machine-learning, neurips2024, outlier-free-quantization, kv-cache-quantization

  OpenDoneDismiss
• highreadme#2

  Add a concise, explicit positioning statement to the README's opening

  Why:

  CURRENT

  # QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

  COPY-PASTE FIX

  Copy

  # QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
  
  This repository provides the official code for QuaRot, a novel method for end-to-end 4-bit quantization of large language models (LLMs).

  OpenDoneDismiss
• mediumreadme#3

  Add a 'Key Features' section to the README

  Why:

  COPY-PASTE FIX

  Copy

  ## Key Features
  
  *   **End-to-End 4-Bit Quantization:** Quantizes all LLM components, including weights, activations, and KV cache, to 4 bits.
  *   **Outlier-Free Inference:** Employs a novel rotation scheme to remove outliers from hidden states, simplifying quantization.
  *   **High Accuracy:** Achieves minimal perplexity loss (0.29 WikiText) and retains 99% zero-shot performance on LLaMa2-70B.
  *   **Computational Invariance:** Applies rotations without altering model output, making quantization robust.
  *   **Optimized Kernels:** Includes custom kernels for high-performance 4-bit operations.

  OpenDoneDismiss

Category GEO backends resolved for this scan: google/gemini-2.5-flash, deepseek/deepseek-v4-flash

• CATEGORY QUERY

  How can I achieve high-performance 4-bit inference for large language models?

  you: not recommended
  AI recommended (in order):

  1. NVIDIA TensorRT-LLM
  2. vLLM
  3. Intel OpenVINO
  4. llama.cpp
  5. GGML/GGUF
  6. Hugging Face Optimum
  7. ONNX Runtime
  8. Intel Neural Compressor
  9. Qualcomm AI Engine Direct (QNN)

  AI recommended 9 alternatives but never named spcl/QuaRot. This is the gap to close.

  Show full AI answer
• CATEGORY QUERY

  What techniques enable full 4-bit quantization of all LLM components, including KV cache?

  you: not recommended
  AI recommended (in order):

  1. GPTQ
  2. AWQ
  3. SmoothQuant
  4. FP8
  5. QLoRA

  AI recommended 5 alternatives but never named spcl/QuaRot. This is the gap to close.

  Show full AI answer

• Metadata completeness

  warn

  Suggestion:

• README presence

  pass

• Compared to common alternatives in this category, what is the core differentiator of spcl/QuaRot?

  pass
  ✓AI named spcl/QuaRot explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• If a team adopts spcl/QuaRot in production, what risks or prerequisites should they evaluate first?

  pass
  ✓AI named spcl/QuaRot explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• In one sentence, what problem does the repo spcl/QuaRot solve, and who is the primary audience?

  pass
  ✓AI named spcl/QuaRot explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?