Bot Insights: Strategic Bot Management for the AI Era

Automated traffic accounted for 51% of web traffic in 2024, according to the cybersecurity firm Imperva in their 2025 Bad Bot Report. 

While organizations that operate online have been dealing with bots since the internet first became widespread, early bots were humble web crawlers, often from search engine companies, or simple content scrapers. Organizations could use basic rules to choose which bots to allow and which to block. The next wave saw a massive rise in spam bots, sophisticated scrapers, bot nets, and more, all of which mask their activity to evade simple bot filtering techniques.

AI bots are the latest category of bots on the web, presenting new challenges and opportunities for organizations. In some cases, they are operated by AI service providers that crawl websites to acquire training data for training large language models (or LLMs). These bots are hungry for content (text and images) from a diverse set of sources to improve the accuracy and reliability of their models. AI bots often have different crawl patterns and request frequencies than traditional search engine crawlers.

Unlike traditional search engine bots that primarily index your content and send traffic back to your site, AI bots can be more challenging to evaluate. In some cases, they scrape content without driving meaningful traffic or revenue. In others, they may indirectly increase visibility by influencing generative engine optimization (GEO), shaping how your brand, products, and expertise appear in AI-generated answers.

This is just the tip of the iceberg. As agentic AI services interact with your web properties on behalf of human users, they can be used to purchase your services, but they can also engage with and recommend competitors, misrepresent your content, and use your content in unintended ways.

Some organizations may opt to block AI bots entirely, some may choose to take a wait-and-see approach, and some want AI bots to crawl their website to ensure their products and services are included in LLM search results. Organizations have to tailor their response strategies to match the intent, sophistication, and impact of each type of bot.

For publishers and content producers, this raises urgent questions: Are these bots eroding your revenue streams? Should you block them entirely or negotiate licensing agreements? How do you distinguish between bots that create value and those that simply extract it?

For executives responsible for bot strategy across any industry, the challenge is broader: What’s actually happening on your web properties? Which bots are accessing what content? What’s the business impact? And how do you craft a policy that balances protection, partnerships, and monetization?

Introducing Bot Insights: Visibility Into Bot Behavior

Hydrolix is now offering Bot Insights, which is designed to surface long-term visibility into bot behavior and bot traffic patterns from your CDN logs. The majority of internet content is served by CDNs, and Bot Insights launches with support for Akamai, Fastly, Cloudflare, Tencent EdgeOne, and Amazon CloudFront logs, which together account for 80% of the world’s CDN traffic.

Our solution complements existing bot manager products, which are often operationally focused and have a limited data window (usually 30 days). How can you develop a thoughtful bot management strategy when you can’t see patterns over time? How do you build consensus among stakeholders when you lack the historical data to support your decisions?

The following dashboard shows how Bot Insights can help you understand valuable metrics about the amount and kind of traffic coming from bots, including total AI bot requests and bot request volume by category.

Bot Insights empowers you to answer critical questions like:

• What percentage of your traffic comes from bots, and how has that changed over time?
• How much of your bot traffic is from AI bots?
• Which AI bot vendors are crawling your properties, and how frequently are they returning?
• What percentage of bot traffic is being served cached content?
• When was the last time a search engine bot (such as Google or Bing) crawled your site?

Most importantly, you can answer these questions for any time period you choose—weeks, months, or even years. You can track trends, validate bot policy changes, and make informed decisions based on comprehensive data.

The bot landscape is incredibly vast, and each bot type and vendor needs to be treated specially to protect core functions and revenue streams while preventing abuse or content theft. You might be interested in blocking most AI bots if you don’t have a licensing agreement with the vendors operating those bots. However, you’ll likely also want to ensure that search engine bots continue to have access.

After some internal discussion with multiple teams, you might procure a short list of bot categories like the following:

Let’s look at a second example, this one for eCommerce. Imagine you’re a key decision maker at a retailer. You start to notice that a growing portion of new and repeat users are originating from LLM products like Perplexity, ChatGPT, and Claude instead of traditional search engines like Google and Bing. You might want to make generative engine optimization (or GEO), which is focused on optimizing web pages for discovery and recommendation by LLM providers, a primary focus for increasing awareness and future revenue streams.

In this situation, you might choose to allow AI bots operated by the most popular LLM providers to freely crawl your website so they can better recommend and direct customers to your products. Here’s what the list of allowed and blocked bot categories might look like.

Using Bot Insights, you can start by understanding existing bot traffic volumes, identifying the bot traffic that is having a negative impact, and ensuring that good bots are still able to receive content from your CDNs.

Get Started With Bot Insights

Bot Insights is a pre-built bundle that includes dashboards and standardized data for five CDNs without any additional configuration needed: Akamai, Cloudflare, AWS CloudFront, Tencent, and Fastly (with more CDN support coming in the near future). Once the solution is deployed, you get immediate access to how different types of bots interact with your CDN infrastructure.

There are three ways to get started:

• If you’re using Hydrolix in the AWS Marketplace, you can deploy the solution automatically through the UI. Select Bot Insights in the Solutions and Data Sources menu to get started. Try a free trial of Hydrolix in the AWS Marketplace.
• If you’re using Akamai TrafficPeak, an observability solution powered by Hydrolix, we can work with you to get Bot Insights deployed in your TrafficPeak instance. Learn more about TrafficPeak.
• If you’re using Hydrolix outside of the AWS Marketplace (such as with bring your own cloud), check in with your account manager to get started with Bot Insights.
• Interested in learning more about Hydrolix? Read more about Bot Insights and try a proof of concept or book a demo.

The AI bot era is here. It’s time to move from reactive blocking to strategic bot management. Bot Insights gives you the visibility you need to make informed decisions that protect your business while capitalizing on legitimate opportunities.

FAQ