FMInference/H2O

• highreadme#1

  Reposition README opening to highlight practical benefit for LLM inference cost/memory

  Why:

  CURRENT

  Code for the paper "**H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models**"
  
  Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen

  COPY-PASTE FIX

  Copy

  H2O significantly reduces the memory footprint and computational cost of Large Language Model (LLM) inference, especially for long-content generation. It achieves this through a novel KV cache eviction policy, the Heavy-Hitter Oracle, which identifies and prioritizes critical tokens. This repository provides the code for our NeurIPS'23 paper: "**H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models**" by Zhenyu Zhang et al.

  OpenDoneDismiss
• highlicense#2

  Add a LICENSE file to the repository

  Why:

  CURRENT

  (no LICENSE file detected — the repo has no recognizable license)

  COPY-PASTE FIX

  Copy

  Create a `LICENSE` file in the repository root containing the text of the MIT License, as suggested by the badge in the README.

  OpenDoneDismiss
• mediumtopics#3

  Expand repository topics to include more specific inference optimization keywords

  Why:

  CURRENT

  gpt-3, heavy-hitters, high-throughput, kv-cache, large-language-models, sparsity

  COPY-PASTE FIX

  Copy

  gpt-3, heavy-hitters, high-throughput, kv-cache, large-language-models, sparsity, llm-inference-optimization, memory-optimization, cost-reduction, generative-ai, deep-learning-inference

  OpenDoneDismiss

Category GEO backends resolved for this scan: google/gemini-2.5-flash, deepseek/deepseek-v4-flash

• CATEGORY QUERY

  How to reduce KV cache memory usage for large language model inference?

  you: not recommended
  AI recommended (in order):

  1. AWQ
  2. GPTQ
  3. bitsandbytes
  4. vLLM
  5. Hugging Face Transformers
  6. Hugging Face `generate()` with `assistant_model`
  7. FlashAttention 2
  8. LongRoPE
  9. StreamingLLM

  AI recommended 9 alternatives but never named FMInference/H2O. This is the gap to close.

  Show full AI answer
• CATEGORY QUERY

  Seeking methods to optimize LLM inference costs for generating extensive text.

  you: not recommended
  AI recommended (in order):

  1. bitsandbytes
  2. AWQ
  3. GPTQ
  4. Google's Speculative Decoding
  5. Medusa
  6. vLLM
  7. Text Generation Inference (TGI)
  8. NVIDIA TensorRT-LLM
  9. Redis
  10. functools.lru_cache

  AI recommended 10 alternatives but never named FMInference/H2O. This is the gap to close.

  Show full AI answer

• Metadata completeness

  warn

  Suggestion:

• README presence

  pass

• Compared to common alternatives in this category, what is the core differentiator of FMInference/H2O?

  pass
  ✓AI named FMInference/H2O explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• If a team adopts FMInference/H2O in production, what risks or prerequisites should they evaluate first?

  pass
  ✓AI named FMInference/H2O explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• In one sentence, what problem does the repo FMInference/H2O solve, and who is the primary audience?

  pass
  ✓AI named FMInference/H2O explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?