REPOGEO REPORT · LITE

jcpeterson/openwebtext

Default branch master · commit 2e3f21d3 · scanned 5/30/2026, 11:33:23 AM

GitHub: 763 stars · 85 forks

AI VISIBILITY SCORE

35 /100

Critical

Category recall

0 / 2

Not recommended in any query

Rule findings

1 pass · 1 warn · 0 fail

Objective metadata checks

AI knows your name

3 / 3

Direct prompts that named your repo

HOW TO READ THIS REPORT

Action plan is what to do next — copy-pasteable changes prioritized by impact. Category visibility is the real GEO test: when a user asks an AI a brand-free question that should surface jcpeterson/openwebtext, does the AI actually recommend you — or your competitors? Objective checks verify the metadata signals AI engines weight first. Self-mention check detects whether AI even knows you exist by name.

Action plan — copy-paste fixes

2 prioritized changes generated by gemini-2.5-flash. Mark items done after you ship the fix.

OVERALL DIRECTION

highreadme#1

Reposition the README's opening to emphasize dataset purpose for LLMs

Why:

CURRENT

# OpenWebText

Joshua Peterson, Stephan Meylan, & David Bourgin

Open clone of OpenAI's unreleased WebText dataset (blog, paper, code) scraper used to train GPT-2. The current result is just over 23 million URLs and over 10 million HTML pages.

COPY-PASTE FIX

# OpenWebText: An Open-Source WebText Dataset for LLM Training

Joshua Peterson, Stephan Meylan, & David Bourgin

OpenWebText is a large, high-quality, open-source text corpus, serving as an alternative to OpenAI's proprietary WebText dataset used to train GPT-2. This project provides both the scraper and pre-filtered URL lists to build a dataset of over 23 million URLs and 10 million HTML pages, ideal for training advanced language models.

mediumreadme#2

Add a 'Why OpenWebText?' section to differentiate from generic tools

Why:

COPY-PASTE FIX

## Why OpenWebText? (Compared to other tools)

Unlike general-purpose web scrapers (like Scrapy or Beautiful Soup) or data processing frameworks (like Apache Spark), OpenWebText is specifically designed to build a high-quality, large-scale text dataset for training language models. It provides a complete pipeline from URL filtering to text extraction, aiming to replicate and openly provide a resource similar to OpenAI's proprietary WebText.

Category GEO backends resolved for this scan: google/gemini-2.5-flash, deepseek/deepseek-v4-flash

Category visibility — the real GEO test

Brand-free queries asked to google/gemini-2.5-flash. Did AI recommend you, or someone else?

Same questions for every model — switch tabs to compare answers and rankings.

Recall

0 / 2

0% of queries surface jcpeterson/openwebtext

Avg rank

—

Lower is better. #1 = top recommendation.

Share of voice

Of all named tools, what % are you?

Top rival

Common Crawl

Recommended in 1 of 2 queries

COMPETITOR LEADERBOARD

Common Crawl · recommended 1×
Apache Spark · recommended 1×
Dask · recommended 1×
justext · recommended 1×
boilerpipe · recommended 1×

CATEGORY QUERY
How to efficiently build a large-scale web text dataset for training advanced language models?
you: not recommended
AI recommended (in order):
1. Common Crawl
2. Apache Spark
3. Dask
4. justext
5. boilerpipe
6. langdetect
7. fastText
8. datasketch
9. BeautifulSoup
10. Scrapy
11. Apache Nutch
12. Newspaper3k
13. trafilatura
AI recommended 13 alternatives but never named jcpeterson/openwebtext. This is the gap to close.
Show full AI answer
CATEGORY QUERY
What are the best tools for scraping and cleaning massive amounts of historical web data?
you: not recommended
AI recommended (in order):
1. Scrapy (scrapy/scrapy)
2. Beautiful Soup 4 (beautifulsoup4/beautifulsoup4)
3. Requests (psf/requests)
4. HTTPX (encode/httpx)
5. Playwright (microsoft/playwright)
6. Selenium (SeleniumHQ/selenium)
7. Apache Nutch (apache/nutch)
8. Pandas (pandas-dev/pandas)
9. OpenRefine (OpenRefine/OpenRefine)
10. Talend Open Studio for Data Integration
AI recommended 10 alternatives but never named jcpeterson/openwebtext. This is the gap to close.
Show full AI answer

Objective checks

Rule-based audits of metadata signals AI engines weight most.

Metadata completeness
warn
Suggestion:
README presence
pass

Self-mention check

Does AI even know your repo exists when asked about it directly?

Compared to common alternatives in this category, what is the core differentiator of jcpeterson/openwebtext?
pass
AI named jcpeterson/openwebtext explicitly
AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?
If a team adopts jcpeterson/openwebtext in production, what risks or prerequisites should they evaluate first?
pass
AI named jcpeterson/openwebtext explicitly
AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?
In one sentence, what problem does the repo jcpeterson/openwebtext solve, and who is the primary audience?
pass
AI named jcpeterson/openwebtext explicitly
AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

Embed your GEO score

Drop this badge into the README of jcpeterson/openwebtext. It auto-updates whenever the report is rescanned and links back to the latest report — easy public proof that you care about AI discoverability.

Live preview

MARKDOWN (README)

[![RepoGEO](https://repogeo.com/badge/jcpeterson/openwebtext.svg)](https://repogeo.com/en/r/jcpeterson/openwebtext)

HTML

<a href="https://repogeo.com/en/r/jcpeterson/openwebtext"><img src="https://repogeo.com/badge/jcpeterson/openwebtext.svg" alt="RepoGEO" /></a>

Pro

Subscribe to Pro for deep diagnoses

jcpeterson/openwebtext — Lite scans stay free; this card itemizes Pro deep limits vs Lite.

Deep reports10 / month
Brand-free category queries5 vs 2 in Lite
Prioritized action items8 vs 3 in Lite