Building a TechCrunch Scraper with Python: RSS First, HTML Second

https3A2F2Fdev-to-uploads.s3.us-east-2.amazonaws.com2Fuploads2Farticles2Ftx7w84fxchn5qaocztlz

TL;DR Use TechCrunch’s public feed for recent article discovery when it satisfies the project; it is simpler and more stable than crawling listing pages. Use article HTML only for fields that the feed does not supply, and keep the original URL and publication metadata. Prefer semantic article markup and JSON-LD over presentation classes. Respect current terms, robots directives, copyright, and rate limits; store only the content needed for the authorized use. Make the pipeline idempotent with canonical URLs, publication identifiers, retrieval timestamps, and content hashes. Why I approached it this way For this kind of news workflow, I start with the public feed and fetch article pages only when the feed does not contain a required field. That keeps discovery simple and…

Continue reading at DEV Community dev.to
Advertisement