👾 Fast and simple video download library and CLI tool written in Go

Elegant Scraper and Crawler Framework for Golang

A next-generation crawling and spidering framework.

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Declarative web scraping

Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes

A powerful browser crawler for web vulnerability scanners

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Gospider - Fast web spider written in Go

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Polite, slim and concurrent web crawler.

Dynamic server-side rendering using headless Chrome

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

The fastest dork scanner written in Go.

一个方便安全研究人员获取每日安全日报的爬虫和推送程序，目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄武、天融信、360等实验室博客，持续更新中。

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

🐾 Creeper - The Next Generation Crawler Framework (Go)