RRepoGEO

REPOGEO REPORT · LITE

openai/SWELancer-Benchmark

Default branch main · commit 4afbde31 · scanned 6/21/2026, 11:20:20 PM

GitHub: 1,440 stars · 137 forks

Scan history for this repo

Score trend below includes all ready runs (older left, newer right; scroll horizontally if needed). The table is collapsed by default—expand for newest-first rows, 10 per page.

Score trend (left → right: older → newer)

2 ready scans. Expand the table below for newest-first rows (10 per page, paginated).

AI VISIBILITY SCORE
18 /100
Critical
Category recall
0 / 2
Not recommended in any query
Rule findings
0 pass · 1 warn · 1 fail
Objective metadata checks
AI knows your name
2 / 3
Direct prompts that named your repo
HOW TO READ THIS REPORT

Action plan is what to do next — copy-pasteable changes prioritized by impact. Category visibility is the real GEO test: when a user asks an AI a brand-free question that should surface openai/SWELancer-Benchmark, does the AI actually recommend you — or your competitors? Objective checks verify the metadata signals AI engines weight first. Self-mention check detects whether AI even knows you exist by name.

Action plan — copy-paste fixes

2 prioritized changes generated by gemini-2.5-flash. Mark items done after you ship the fix.

OVERALL DIRECTION
  • highreadme#1
    Reposition README to clarify this repo's purpose as the benchmark dataset and paper code

    Why:

    CURRENT
    # SWELancer

The SWE-Lancer codebase has been merged into https://github.com/openai/preparedness! 

**Please see https://github.com/openai/preparedness to run SWELancer**.
    COPY-PASTE FIX
    # SWELancer: Dataset and Code for the SWE-Lancer Benchmark Paper

This repository contains the dataset and original code for the SWE-Lancer paper: "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?". It provides a benchmark for evaluating large language models on complex, real-world software engineering tasks.

The active codebase for running SWELancer has been merged into https://github.com/openai/preparedness. Please see that repository to run SWELancer.
  • highlicense#2
    Add a LICENSE file to the repository

    Why:

    COPY-PASTE FIX
    Add a `LICENSE` file (e.g., `LICENSE.md`) to the repository root, containing the text of the Apache-2.0 License.

Category GEO backends resolved for this scan: google/gemini-2.5-flash, deepseek/deepseek-v4-flash

Category visibility — the real GEO test

Brand-free queries asked to google/gemini-2.5-flash. Did AI recommend you, or someone else?

Same questions for every model — switch tabs to compare answers and rankings.

Recall
0 / 2
0% of queries surface openai/SWELancer-Benchmark
Avg rank
—
Lower is better. #1 = top recommendation.
Share of voice
0%
Of all named tools, what % are you?
Top rival
SWE-bench
Recommended in 2 of 2 queries
COMPETITOR LEADERBOARD
  1. SWE-bench · recommended 2×
  2. HumanEval · recommended 2×
  3. CodeXGLUE · recommended 2×
  4. HumanEval-X · recommended 1×
  5. MultiPL-E · recommended 1×
  • CATEGORY QUERY
    How to benchmark large language models on complex software development challenges?
    you: not recommended
    AI recommended (in order):
    1. SWE-bench
    2. HumanEval
    3. HumanEval-X
    4. MultiPL-E
    5. CodeContests
    6. BigCodeBench
    7. CodeXGLUE

    AI recommended 7 alternatives but never named openai/SWELancer-Benchmark. This is the gap to close.

    Show full AI answer
  • CATEGORY QUERY
    What are the best datasets for assessing AI performance in practical software engineering roles?
    you: not recommended
    AI recommended (in order):
    1. HumanEval
    2. MBPP
    3. CodeXGLUE
    4. CONCODE
    5. Defects4J
    6. QuixBugs
    7. APPS
    8. SWE-bench
    9. CodeSearchNet
    10. The Stack

    AI recommended 10 alternatives but never named openai/SWELancer-Benchmark. This is the gap to close.

    Show full AI answer

Objective checks

Rule-based audits of metadata signals AI engines weight most.

  • Metadata completeness
    fail

    Suggestion:

  • README presence
    warn

    Suggestion:

Self-mention check

Does AI even know your repo exists when asked about it directly?

  • Compared to common alternatives in this category, what is the core differentiator of openai/SWELancer-Benchmark?
    pass
    AI did not name openai/SWELancer-Benchmark — likely talking about a different project

    AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

  • If a team adopts openai/SWELancer-Benchmark in production, what risks or prerequisites should they evaluate first?
    pass
    AI named openai/SWELancer-Benchmark explicitly

    AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

  • In one sentence, what problem does the repo openai/SWELancer-Benchmark solve, and who is the primary audience?
    pass
    AI named openai/SWELancer-Benchmark explicitly

    AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

Embed your GEO score

Drop this badge into the README of openai/SWELancer-Benchmark. It auto-updates whenever the report is rescanned and links back to the latest report — easy public proof that you care about AI discoverability.

RepoGEO badge previewLive preview
MARKDOWN (README)
[![RepoGEO](https://repogeo.com/badge/openai/SWELancer-Benchmark.svg)](https://repogeo.com/en/r/openai/SWELancer-Benchmark)
HTML
<a href="https://repogeo.com/en/r/openai/SWELancer-Benchmark"><img src="https://repogeo.com/badge/openai/SWELancer-Benchmark.svg" alt="RepoGEO" /></a>
Pro

Subscribe to Pro for deep diagnoses

openai/SWELancer-Benchmark — Lite scans stay free; this card itemizes Pro deep limits vs Lite.

  • Deep reports10 / month
  • Brand-free category queries5 vs 2 in Lite
  • Prioritized action items8 vs 3 in Lite