[webconnectivity]: Support user-agent strings in request headers to bypass scrapper security #1741

Activity

1. DecFox

   self-assigned this

   on Sep 18, 2025
2. DecFox

   added the

   Feature

   issue type on Sep 18, 2025
3. DecFox

   added

   enhancementNew feature request or improvement to existing functionalityNew feature request or improvement to existing functionality

   on Sep 18, 2025
4. DecFox commented on Sep 18, 2025

   

   DecFox

   on Sep 18, 2025

   ContributorAuthor

   More actions

   @hellais pointed out that some work on the scrapper invasion strategies in the past reflected that WAFs will also match the TLS stack to the user-agent strings and consider the ordering of HTTP headers in determining what to do. The go stdlib does not support maintaining the order of request headers but we should in my opinion still support user-agent strings in the experiment as a start. These are the headers that should be used for Chrome 132:

   // Updated headers for Chrome 132
   	headers.Add("sec-ch-device-memory", "8")
   	headers.Add("sec-ch-ua", `"Not A(Brand";v="8", "Chromium";v="132", "Google Chrome";v="132"`)
   	headers.Add("sec-ch-ua-mobile", "?0")
   	headers.Add("sec-ch-ua-arch", `"x86"`)
   	headers.Add("sec-ch-ua-platform", `"Linux"`)
   	headers.Add("sec-ch-ua-model", `""`)
   	headers.Add("sec-ch-ua-full-version-list", `"Not A(Brand";v="8.0.0.0", "Chromium";v="132.0.6834.159", "Google Chrome";v="132.0.6834.159"`)
   	headers.Add("dnt", "1")
   	headers.Add("upgrade-insecure-requests", "1")
   	headers.Add("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36")
   	headers.Add("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7")
   	headers.Add("sec-fetch-site", "none")
   	headers.Add("sec-fetch-mode", "navigate")
   	headers.Add("sec-fetch-user", "?1")
   	headers.Add("sec-fetch-dest", "document")
   	headers.Add("accept-encoding", "gzip, deflate, br, zstd")
   	headers.Add("accept-language", "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,sv;q=0.6")
   	headers.Add("priority", "u=0, i")
   

   👍React with 👍1Reacted by Arturo Filastò
5. DecFox

   changed the title [-][webconnectivity]: Support user-agent strings in request headers to bypass by scrapper security[/-] on Sep 18, 2025
6. hellais

   added

   feature

   and removed

   enhancementNew feature request or improvement to existing functionalityNew feature request or improvement to existing functionality

   on Aug 10, 2026