RRepoGEO

REPOGEO REPORT · LITE

togethercomputer/RedPajama-Data

Default branch main · commit 6d2cee9d · scanned 5/29/2026, 1:58:41 PM

GitHub: 4,945 stars · 372 forks

AI VISIBILITY SCORE
53 /100
Needs work
Category recall
1 / 2
Avg rank #6.0 when recommended
Rule findings
1 pass · 1 warn · 0 fail
Objective metadata checks
AI knows your name
3 / 3
Direct prompts that named your repo
HOW TO READ THIS REPORT

Action plan is what to do next — copy-pasteable changes prioritized by impact. Category visibility is the real GEO test: when a user asks an AI a brand-free question that should surface togethercomputer/RedPajama-Data, does the AI actually recommend you — or your competitors? Objective checks verify the metadata signals AI engines weight first. Self-mention check detects whether AI even knows you exist by name.

Action plan — copy-paste fixes

2 prioritized changes generated by gemini-2.5-flash. Mark items done after you ship the fix.

OVERALL DIRECTION
  • highreadme#1
    Clarify the README's opening sentence to emphasize the dataset and its specific codebase

    Why:

    CURRENT
    This repository contains the code for the RedPajama-V2 dataset. For more information on the dataset, check out our blog post. The dataset is also available on HuggingFace. For the code used for the RedPajama-1T dataset, please refer to the `rp_v1` branch in this repo.
    COPY-PASTE FIX
    This repository provides the **RedPajama-V2 dataset itself**, a massive, open-source collection of 30 trillion tokens for training large language models, **along with the full codebase used to construct it**. For more information on the dataset, check out our blog post. The dataset is also available on HuggingFace. For the code used for the RedPajama-1T dataset, please refer to the `rp_v1` branch in this repo.
  • mediumhomepage#2
    Add a homepage URL to the repository's About section

    Why:

    COPY-PASTE FIX
    https://www.together.ai/blog/redpajama-v2

Category GEO backends resolved for this scan: google/gemini-2.5-flash, deepseek/deepseek-v4-flash

Category visibility — the real GEO test

Brand-free queries asked to google/gemini-2.5-flash. Did AI recommend you, or someone else?

Same questions for every model — switch tabs to compare answers and rankings.

Recall
1 / 2
50% of queries surface togethercomputer/RedPajama-Data
Avg rank
#6.0
Lower is better. #1 = top recommendation.
Share of voice
3%
Of all named tools, what % are you?
Top rival
Common Crawl
Recommended in 2 of 2 queries
COMPETITOR LEADERBOARD
  1. Common Crawl · recommended 2×
  2. The Pile · recommended 2×
  3. Scrapy · recommended 1×
  4. Beautiful Soup · recommended 1×
  5. lxml · recommended 1×
  • CATEGORY QUERY
    What are methods for creating high-quality, large-scale datasets for LLM pre-training?
    you: not recommended
    AI recommended (in order):
    1. Common Crawl
    2. Scrapy
    3. Beautiful Soup
    4. lxml
    5. arXiv
    6. JSTOR
    7. New York Times
    8. The Guardian
    9. langdetect
    10. fastText
    11. The Pile
    12. PubMed Central
    13. GitHub
    14. Wikipedia
    15. BooksCorpus
    16. Project Gutenberg
    17. OpenWebText2
    18. GPT-3.5
    19. GPT-4
    20. ClinicalTrials.gov
    21. SEC Filings
    22. EDGAR database
    23. LexisNexis
    24. Westlaw
    25. Amazon Mechanical Turk
    26. Scale AI
    27. Appen
    28. datasketch
    29. spaCy
    30. NLTK

    AI recommended 30 alternatives but never named togethercomputer/RedPajama-Data. This is the gap to close.

    Show full AI answer
  • CATEGORY QUERY
    Where can I find open-source, multi-terabyte text datasets for training new LLMs?
    you: #6
    AI recommended (in order):
    1. The Pile
    2. Common Crawl
    3. ccnet
    4. OSCAR (Open Super-large Crawled ALMAnaC coRpus)
    5. C4 (Colossal Clean Crawled Corpus)
    6. RedPajama-Data ← you
    7. Pushshift Reddit Dataset
    8. BookCorpus
    Show full AI answer

Objective checks

Rule-based audits of metadata signals AI engines weight most.

  • Metadata completeness
    warn

    Suggestion:

  • README presence
    pass

Self-mention check

Does AI even know your repo exists when asked about it directly?

  • Compared to common alternatives in this category, what is the core differentiator of togethercomputer/RedPajama-Data?
    pass
    AI named togethercomputer/RedPajama-Data explicitly

    AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

  • If a team adopts togethercomputer/RedPajama-Data in production, what risks or prerequisites should they evaluate first?
    pass
    AI named togethercomputer/RedPajama-Data explicitly

    AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

  • In one sentence, what problem does the repo togethercomputer/RedPajama-Data solve, and who is the primary audience?
    pass
    AI named togethercomputer/RedPajama-Data explicitly

    AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

Embed your GEO score

Drop this badge into the README of togethercomputer/RedPajama-Data. It auto-updates whenever the report is rescanned and links back to the latest report — easy public proof that you care about AI discoverability.

RepoGEO badge previewLive preview
MARKDOWN (README)
[![RepoGEO](https://repogeo.com/badge/togethercomputer/RedPajama-Data.svg)](https://repogeo.com/en/r/togethercomputer/RedPajama-Data)
HTML
<a href="https://repogeo.com/en/r/togethercomputer/RedPajama-Data"><img src="https://repogeo.com/badge/togethercomputer/RedPajama-Data.svg" alt="RepoGEO" /></a>
Pro

Subscribe to Pro for deep diagnoses

togethercomputer/RedPajama-Data — Lite scans stay free; this card itemizes Pro deep limits vs Lite.

  • Deep reports10 / month
  • Brand-free category queries5 vs 2 in Lite
  • Prioritized action items8 vs 3 in Lite