Web Scraper

Web scraper written in Go that uses chromedp (headless Chrome) to crawl websites and convert content to Markdown.

• Go 1.26+
• Google Chrome / Chromium (required by chromedp for headless execution)

cmd/scraper/main.go     CLI (Cobra): 2 commands, `fetch` and `crawl`
internal/crawler/       Uses chromedp to fetch real HTML (with scrolling for lazy-loaded content)
internal/parser/        Filters content by selector (include/exclude)
internal/converter/     HTML → Markdown
internal/storage/       Saves .md files by domain into output/
internal/config/        Configuration (config.yaml or defaults)

1. Build:

go mod tidy
go build -o scraper ./cmd/scraper

2. Create an optional config.yaml in the project root:

output_dir: "./output"
headless: true
timeout: 30s
rate_limit: 1s
user_agent: "Mozilla/5.0 ..."
selectors:
  include: ["main", ".content", "article", "section", "body"]
  exclude: ["nav", "footer", ".sidebar", "script", "style", "iframe", "header", "aside"]

If the file is omitted, the default configuration is used (output_dir: ./output).

3. Run the scraper:

# Fetch one page
./scraper fetch https://example.com

# Fetch from a file (note: the current code reads the entire file as one string, so it only works with a single-line file)
./scraper fetch -i urls.txt

# Crawl the entire site by depth
./scraper crawl https://example.com --depth 2

Results are saved as output/<domain>/<path>.md with frontmatter.

• The fetch -i command currently reads the entire file as a single URL string, so it should only be used when the file contains one URL.
• Google Chrome / Chromium must be installed before running the scraper.