je-suis-tm/web-scraping

• highreadme#1

  Reposition the README's opening to emphasize "tutorials" and "site-specific examples"

  Why:

  CURRENT

  My understanding of web scraping is patience and attention to details. Scraping is not rocket science (deep learning is). When I do scraping, I typically spend 50% of my time in analyzing the source (navigate through HTML parse tree or inspect element to find the post form) and the rest 50% in ETL. The most useful tools for me are `requests`, `bs4` and `re`. Some people may recommend `selenium` for non-static website. To be honest, I have never used `selenium` throughout my career, but dynamic websites like Facebook and Twitter are still within my grasp. You see? patience and attention to details matter. This repository contains a couple of python web scrapers. These scrapers mainly target at different commodity future exchanges and influential media websites (or so-called fake news, lol). Most scripts were written during my early days of Python learning. Since this repository gained unexpected popularity, I have restructured everything to make it more user-friendly. All the scripts featured in this repository are ready for use. Each script is designed to feature a unique technique that I found useful throughout my experience of data engineering.

  COPY-PASTE FIX

  Copy

  This repository provides detailed Python web scraping tutorials and ready-to-use, site-specific examples. It focuses on extracting financial market data from sources like Reddit WallStreetBets, CME, US Treasury, CFTC, LME, MacroTrends, and SHFE, alongside news and alternative data from major media outlets such as Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, and The Economist. Each script demonstrates unique techniques using `requests`, `bs4`, and `re`, including methods for dynamic websites without relying on `selenium`.

  OpenDoneDismiss
• mediumtopics#2

  Add topics that highlight the tutorial/example nature and the 'no-selenium' approach

  Why:

  CURRENT

  bloomberg, data-scraper, data-scraping, financial-data, financial-times, futures, futures-historical-data, news-scraper, news-websites, newsletter, options-data, python-web-scraper, reuters, scrapper, sraping, wall-street-journal, wallstreetbets, web-scraper, web-scrapers, web-scraping

  COPY-PASTE FIX

  Copy

  bloomberg, data-scraper, data-scraping, financial-data, financial-times, futures, futures-historical-data, news-scraper, news-websites, newsletter, options-data, python-web-scraper, reuters, scrapper, sraping, wall-street-journal, wallstreetbets, web-scraper, web-scrapers, web-scraping, web-scraping-tutorials, python-web-scraping-examples, no-selenium-scraping, dynamic-web-scraping

  OpenDoneDismiss
• lowabout#3

  Emphasize 'practical examples' in the description

  Why:

  CURRENT

  Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

  COPY-PASTE FIX

  Copy

  Detailed web scraping tutorials and practical, site-specific examples for dummies, featuring financial data crawlers on Reddit WallStreetBets, CME (options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE, and alternative data crawlers for Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist.

  OpenDoneDismiss

Category GEO backends resolved for this scan: google/gemini-2.5-flash, deepseek/deepseek-v4-flash

• CATEGORY QUERY

  How to scrape financial market data and news articles using Python?

  you: not recommended
  AI recommended (in order):

  1. yfinance
  2. Beautiful Soup
  3. Scrapy
  4. Alpha Vantage API
  5. NewsAPI.org
  6. finnhub.io API
  7. Selenium

  AI recommended 7 alternatives but never named je-suis-tm/web-scraping. This is the gap to close.

  Show full AI answer
• CATEGORY QUERY

  Looking for Python web scraping tutorials to extract data from dynamic websites without Selenium.

  you: not recommended
  AI recommended (in order):

  1. Beautiful Soup (crummy/BeautifulSoup)
  2. requests-html (psf/requests-html)
  3. Playwright (microsoft/playwright)
  4. Scrapy (scrapy/scrapy)
  5. Scrapy-Splash (scrapy-plugins/scrapy-splash)
  6. Scrapy-Playwright (scrapy-plugins/scrapy-playwright)
  7. Web Scraping with Python by Ryan Mitchell
  8. Real Python
  9. YouTube

  AI recommended 9 alternatives but never named je-suis-tm/web-scraping. This is the gap to close.

  Show full AI answer

• Metadata completeness

  pass

• README presence

  pass

• Compared to common alternatives in this category, what is the core differentiator of je-suis-tm/web-scraping?

  pass
  ✓AI named je-suis-tm/web-scraping explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• If a team adopts je-suis-tm/web-scraping in production, what risks or prerequisites should they evaluate first?

  pass
  ✓AI named je-suis-tm/web-scraping explicitly

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?

• In one sentence, what problem does the repo je-suis-tm/web-scraping solve, and who is the primary audience?

  pass
  ⚠AI did not name je-suis-tm/web-scraping — likely talking about a different project

  AI answers can be confidently wrong. Read for accuracy: does it match your actual tech stack, audience, and differentiator?