je-suis-tm/web-scraping

• highreadme#1

  Reposition the README's opening to emphasize "tutorials" and "site-specific examples"

  原因：

  当前

  My understanding of web scraping is patience and attention to details. Scraping is not rocket science (deep learning is). When I do scraping, I typically spend 50% of my time in analyzing the source (navigate through HTML parse tree or inspect element to find the post form) and the rest 50% in ETL. The most useful tools for me are `requests`, `bs4` and `re`. Some people may recommend `selenium` for non-static website. To be honest, I have never used `selenium` throughout my career, but dynamic websites like Facebook and Twitter are still within my grasp. You see? patience and attention to details matter. This repository contains a couple of python web scrapers. These scrapers mainly target at different commodity future exchanges and influential media websites (or so-called fake news, lol). Most scripts were written during my early days of Python learning. Since this repository gained unexpected popularity, I have restructured everything to make it more user-friendly. All the scripts featured in this repository are ready for use. Each script is designed to feature a unique technique that I found useful throughout my experience of data engineering.

  复制粘贴的修复

  复制

  This repository provides detailed Python web scraping tutorials and ready-to-use, site-specific examples. It focuses on extracting financial market data from sources like Reddit WallStreetBets, CME, US Treasury, CFTC, LME, MacroTrends, and SHFE, alongside news and alternative data from major media outlets such as Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, and The Economist. Each script demonstrates unique techniques using `requests`, `bs4`, and `re`, including methods for dynamic websites without relying on `selenium`.

  待办完成忽略
• mediumtopics#2

  Add topics that highlight the tutorial/example nature and the 'no-selenium' approach

  原因：

  当前

  bloomberg, data-scraper, data-scraping, financial-data, financial-times, futures, futures-historical-data, news-scraper, news-websites, newsletter, options-data, python-web-scraper, reuters, scrapper, sraping, wall-street-journal, wallstreetbets, web-scraper, web-scrapers, web-scraping

  复制粘贴的修复

  复制

  bloomberg, data-scraper, data-scraping, financial-data, financial-times, futures, futures-historical-data, news-scraper, news-websites, newsletter, options-data, python-web-scraper, reuters, scrapper, sraping, wall-street-journal, wallstreetbets, web-scraper, web-scrapers, web-scraping, web-scraping-tutorials, python-web-scraping-examples, no-selenium-scraping, dynamic-web-scraping

  待办完成忽略
• lowabout#3

  Emphasize 'practical examples' in the description

  原因：

  当前

  Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

  复制粘贴的修复

  复制

  Detailed web scraping tutorials and practical, site-specific examples for dummies, featuring financial data crawlers on Reddit WallStreetBets, CME (options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE, and alternative data crawlers for Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist.

  待办完成忽略

本次扫描解析到的品类 GEO 通道：google/gemini-2.5-flash, deepseek/deepseek-v4-flash

• 品类问题

  How to scrape financial market data and news articles using Python?

  你：未被推荐
  AI 推荐顺序：

  1. yfinance
  2. Beautiful Soup
  3. Scrapy
  4. Alpha Vantage API
  5. NewsAPI.org
  6. finnhub.io API
  7. Selenium

  AI 推荐了 7 个替代方案，却始终没点名 je-suis-tm/web-scraping。这就是要补上的差距。

  查看 AI 完整回答
• 品类问题

  Looking for Python web scraping tutorials to extract data from dynamic websites without Selenium.

  你：未被推荐
  AI 推荐顺序：

  1. Beautiful Soup (crummy/BeautifulSoup)
  2. requests-html (psf/requests-html)
  3. Playwright (microsoft/playwright)
  4. Scrapy (scrapy/scrapy)
  5. Scrapy-Splash (scrapy-plugins/scrapy-splash)
  6. Scrapy-Playwright (scrapy-plugins/scrapy-playwright)
  7. Web Scraping with Python by Ryan Mitchell
  8. Real Python
  9. YouTube

  AI 推荐了 9 个替代方案，却始终没点名 je-suis-tm/web-scraping。这就是要补上的差距。

  查看 AI 完整回答

• Metadata completeness

  pass

• README presence

  pass

• Compared to common alternatives in this category, what is the core differentiator of je-suis-tm/web-scraping?

  pass
  ✓AI 明确点名了 je-suis-tm/web-scraping

  AI 的回答可能信誓旦旦却是错的。请按事实核对：技术栈、目标人群、差异化点是不是和你实际的对得上？

• If a team adopts je-suis-tm/web-scraping in production, what risks or prerequisites should they evaluate first?

  pass
  ✓AI 明确点名了 je-suis-tm/web-scraping

  AI 的回答可能信誓旦旦却是错的。请按事实核对：技术栈、目标人群、差异化点是不是和你实际的对得上？

• In one sentence, what problem does the repo je-suis-tm/web-scraping solve, and who is the primary audience?

  pass
  ⚠AI 未点名 je-suis-tm/web-scraping —— 很可能在说另一个项目

  AI 的回答可能信誓旦旦却是错的。请按事实核对：技术栈、目标人群、差异化点是不是和你实际的对得上？