microsoft/onnxruntime-genai

• hightopics#1

  Add relevant topics to the repository

  Why:

  COPY-PASTE FIX

  Copy

  llm, generative-ai, onnx, inference, kv-cache, language-models, ai-framework, on-device-ai, microsoft

  OpenDoneDismiss
• highreadme#2

  Reposition the README's opening to clarify its role as a full GenAI inference framework

  Why:

  CURRENT

  # ONNX Runtime GenAI
  
  ## Status
  
  [](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntimeGenAI.Managed/absoluteLatest)
  
  [](https://github.com/microsoft/onnxruntime-genai/actions/workflows/linux-cpu-x64-nightly-build.yml)
  
  ## Description
  
  Run generative AI models with ONNX Runtime. This API gives you an easy, flexible and performant way of running LLMs on device. It implements the generative AI loop for ONNX models, including pre and post processing, inference with ONNX Runtime, logits processing, search and sampling, KV cache management, and grammar specification for tool calling.

  COPY-PASTE FIX

  Copy

  # ONNX Runtime GenAI
  
  ONNX Runtime GenAI is a high-performance library for running the full generative AI inference loop on-device, building on ONNX Runtime. It provides an easy, flexible, and performant API for large language models (LLMs), including pre/post-processing, KV cache management, search, and sampling.
  
  ## Status
  
  [](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntimeGenAI.Managed/absoluteLatest)
  
  [](https://github.com/microsoft/onnxruntime-genai/actions/workflows/linux-cpu-x64-nightly-build.yml)
  
  ## Description
  
  Run generative AI models with ONNX Runtime. This API gives you an easy, flexible and performant way of running LLMs on device. It implements the generative AI loop for ONNX models, including pre and post processing, inference with ONNX Runtime, logits processing, search and sampling, KV cache management, and grammar specification for tool calling.

  OpenDoneDismiss
• mediumhomepage#3

  Add a homepage URL to the repository metadata

  Why:

  COPY-PASTE FIX

  Copy

  [Link to official documentation or project landing page, e.g., https://onnxruntime.ai/docs/genai/]

  OpenDoneDismiss

Category GEO backends resolved for this scan: google/gemini-2.5-flash, deepseek/deepseek-v4-flash

• CATEGORY QUERY

  Seeking a performant solution for deploying large language models on-device using ONNX.

  you: not recommended
  AI recommended (in order):

  1. ONNX Runtime
  2. OpenVINO
  3. Core ML
  4. TensorRT
  5. Qualcomm AI Engine Direct (QNN)
  6. MediaPipe

  AI recommended 6 alternatives but never named microsoft/onnxruntime-genai. This is the gap to close.

  Show full AI answer
• CATEGORY QUERY

  What tools simplify local inference and KV cache management for generative AI models?

  you: not recommended
  AI recommended (in order):

  1. vLLM (vllm-project/vllm)
  2. llama.cpp (ggerganov/llama.cpp)
  3. Hugging Face Transformers (huggingface/transformers)
  4. TGI (huggingface/text-generation-inference)
  5. TensorRT-LLM (NVIDIA/TensorRT-LLM)
  6. Ollama (ollama/ollama)

  AI recommended 6 alternatives but never named microsoft/onnxruntime-genai. This is the gap to close.

  Show full AI answer

• Metadata completeness

  warn

  Suggestion:

• README presence

  pass

• Compared to common alternatives in this category, what is the core differentiator of microsoft/onnxruntime-genai?

  pass
  ✓AI named microsoft/onnxruntime-genai explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• If a team adopts microsoft/onnxruntime-genai in production, what risks or prerequisites should they evaluate first?

  pass
  ✓AI named microsoft/onnxruntime-genai explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• In one sentence, what problem does the repo microsoft/onnxruntime-genai solve, and who is the primary audience?

  pass
  ✓AI named microsoft/onnxruntime-genai explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?