RRepoGEO

REPOGEO REPORT · LITE

open-compass/VLMEvalKit

Default branch main · commit 0bfa830f · scanned 7/1/2026, 2:31:41 AM

GitHub: 4,245 stars · 726 forks

Scan history for this repo

Score trend below includes all ready runs (older left, newer right; scroll horizontally if needed). The table is collapsed by default—expand for newest-first rows, 10 per page.

Score trend (left → right: older → newer)

2 ready scans. Expand the table below for newest-first rows (10 per page, paginated).

AI VISIBILITY SCORE
58 /100
Needs work
Category recall
1 / 2
Avg rank #14.0 when recommended
Rule findings
2 pass · 0 warn · 0 fail
Objective metadata checks
AI knows your name
3 / 3
Direct prompts that named your repo
HOW TO READ THIS REPORT

Action plan is what to do next — copy-pasteable changes prioritized by impact. Category visibility is the real GEO test: when a user asks an AI a brand-free question that should surface open-compass/VLMEvalKit, does the AI actually recommend you — or your competitors? Objective checks verify the metadata signals AI engines weight first. Self-mention check detects whether AI even knows you exist by name.

Action plan — copy-paste fixes

3 prioritized changes generated by gemini-2.5-flash. Mark items done after you ship the fix.

OVERALL DIRECTION
  • highreadme#1
    Clarify the README's opening statement to explicitly position VLMEvalKit as a toolkit, not just a collection of benchmarks.

    Why:

    CURRENT
    <b>A Toolkit for Evaluating Large Vision-Language Models. </b>
    COPY-PASTE FIX
    <b>VLMEvalKit is the definitive open-source toolkit for systematically evaluating Large Vision-Language Models (LVLMs). It provides a unified framework to benchmark 220+ LVLMs across 80+ datasets, eliminating the need to manage individual benchmarks.</b>
  • mediumtopics#2
    Add more specific topics to improve categorization as an LMM benchmarking toolkit.

    Why:

    CURRENT
    chatgpt, claude, clip, computer-vision, evaluation, gemini, gpt, gpt-4v, gpt4, large-language-models, llava, llm, multi-modal, openai, openai-api, pytorch, qwen, vit, vqa
    COPY-PASTE FIX
    chatgpt, claude, clip, computer-vision, evaluation, gemini, gpt, gpt-4v, gpt4, large-language-models, llava, llm, multi-modal, openai, openai-api, pytorch, qwen, vit, vqa, llm-benchmarking, vlm-evaluation, multimodal-evaluation, evaluation-framework
  • lowreadme#3
    Add a 'Comparison with Alternatives' section to the README.

    Why:

    COPY-PASTE FIX
    ## 💡 Comparison with Alternatives

While general evaluation frameworks like EleutherAI/lm-evaluation-harness or MMEval offer broad benchmarking capabilities, VLMEvalKit specializes in the unique challenges of Large Vision-Language Models. We provide out-of-the-box support for 220+ LVLMs and 80+ benchmarks, focusing on the specific data formats, inference pipelines, and evaluation metrics required for multi-modal AI, offering a more streamlined and comprehensive solution for LMM developers and researchers.

Category GEO backends resolved for this scan: google/gemini-2.5-flash, deepseek/deepseek-v4-flash

Category visibility — the real GEO test

Brand-free queries asked to google/gemini-2.5-flash. Did AI recommend you, or someone else?

Same questions for every model — switch tabs to compare answers and rankings.

Recall
1 / 2
50% of queries surface open-compass/VLMEvalKit
Avg rank
#14.0
Lower is better. #1 = top recommendation.
Share of voice
4%
Of all named tools, what % are you?
Top rival
VQA-v2
Recommended in 1 of 2 queries
COMPETITOR LEADERBOARD
  1. VQA-v2 · recommended 1×
  2. GQA · recommended 1×
  3. OK-VQA · recommended 1×
  4. COCO Captions · recommended 1×
  5. Flickr30k · recommended 1×
  • CATEGORY QUERY
    How can I systematically evaluate the performance of different large vision-language models?
    you: #14
    AI recommended (in order):
    1. VQA-v2
    2. GQA
    3. OK-VQA
    4. COCO Captions
    5. Flickr30k
    6. RefCOCO/RefCOCO+/RefCOCOg
    7. ScienceQA
    8. MM-Vet
    9. POPE
    10. FairFace
    11. ImageNet-A/ImageNet-R/ImageNet-Sketch
    12. OpenAI Evals
    13. Hugging Face Evaluate library
    14. VLMEvalKit ← you
    15. Amazon Mechanical Turk
    16. Scale AI
    17. Appen
    Show full AI answer
  • CATEGORY QUERY
    Looking for an open-source toolkit to benchmark various multi-modal AI models effectively.
    you: not recommended
    AI recommended (in order):
    1. EleutherAI/lm-evaluation-harness (EleutherAI/lm-evaluation-harness)
    2. MMEval
    3. Hugging Face Evaluate
    4. TorchMetrics
    5. MMDetection
    6. MMSegmentation
    7. MMClassification

    AI recommended 7 alternatives but never named open-compass/VLMEvalKit. This is the gap to close.

    Show full AI answer

Objective checks

Rule-based audits of metadata signals AI engines weight most.

  • Metadata completeness
    pass

  • README presence
    pass

Self-mention check

Does AI even know your repo exists when asked about it directly?

  • Compared to common alternatives in this category, what is the core differentiator of open-compass/VLMEvalKit?
    pass
    AI named open-compass/VLMEvalKit explicitly

    AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

  • If a team adopts open-compass/VLMEvalKit in production, what risks or prerequisites should they evaluate first?
    pass
    AI named open-compass/VLMEvalKit explicitly

    AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

  • In one sentence, what problem does the repo open-compass/VLMEvalKit solve, and who is the primary audience?
    pass
    AI named open-compass/VLMEvalKit explicitly

    AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

Embed your GEO score

Drop this badge into the README of open-compass/VLMEvalKit. It auto-updates whenever the report is rescanned and links back to the latest report — easy public proof that you care about AI discoverability.

RepoGEO badge previewLive preview
MARKDOWN (README)
[![RepoGEO](https://repogeo.com/badge/open-compass/VLMEvalKit.svg)](https://repogeo.com/en/r/open-compass/VLMEvalKit)
HTML
<a href="https://repogeo.com/en/r/open-compass/VLMEvalKit"><img src="https://repogeo.com/badge/open-compass/VLMEvalKit.svg" alt="RepoGEO" /></a>
Pro

Subscribe to Pro for deep diagnoses

open-compass/VLMEvalKit — Lite scans stay free; this card itemizes Pro deep limits vs Lite.

  • Deep reports10 / month
  • Brand-free category queries5 vs 2 in Lite
  • Prioritized action items8 vs 3 in Lite
open-compass/VLMEvalKit — RepoGEO report