GitHub - daniel-maxwell/WebCrawler: Multi threaded Go-based crawler with per-domain limits, queue-based scheduling, bloom filter deduplication, robots.txt compliance, user-agent rotation and disaster recovery. This project is under active development, README coming soon! For now, simply clone the project and run main.go to use it.

daniel-maxwell / WebCrawler Public

Notifications You must be signed in to change notification settings
Fork 0
Star 1

Multi threaded Go-based crawler with per-domain limits, queue-based scheduling, bloom filter deduplication, robots.txt compliance, user-agent rotation and disaster recovery. This project is under active development, README coming soon! For now, simply clone the project and run main.go to use it.