en.Wedoany.com Reported - Cloudflare has launched new AI crawler traffic management controls, allowing website owners to define access policies for automated programs based on three functional categories: search, proxy, and training. This feature is now available to all Cloudflare customers, including those on the free tier.
Content owners want to protect their creative work and receive fair compensation for it. Cloudflare Product Manager Jin-He Lee and Product Director Bryan Becker stated that completely blocking content is not a one-size-fits-all solution, and website owners need more flexible options rather than simply "blocking all automated programs at all times."
The company has expanded bot control beyond the simple allow-or-block model. Instead of merely classifying bots as AI or non-AI, they are now categorized by function into search, proxy, and training types, allowing website owners to apply different policies to each category.
The updated framework considers how bots use content after crawling it and encourages providers to use separate crawlers for different functions to improve transparency and access control capabilities.
Starting September 15, 2026, default settings for new domains will change. On pages displaying advertisements, training and proxy crawlers will be blocked by default, while search crawlers will remain allowed by default.
Multi-purpose crawlers that perform both search and training functions will be evaluated based on both policies. If a website blocks training crawlers, multi-purpose crawlers (such as Googlebot, Applebot, and BingBot) will also be blocked, even if search crawlers are allowed. This change will take effect on September 15, 2026. Website owners who do not wish to adopt the new default settings can opt out through security settings before September 15, retaining the current behavior for training crawlers performing search functions. Cloudflare stated it will notify customers before the change so they have time to review and update their settings.
Cloudflare has also launched BotBase, a searchable database of known bots, including verified bots and AI agents. This database provides bot management customers with a centralized view based on Cloudflare's new taxonomy, displaying the classification and behavior of verified bots. Administrators can use BotBase to browse the verified bot directory, search for specific bots, view classifications, filter traffic by individual bots, and copy detection IDs for use in security rules. The company plans to add more control features later this year, making it easier for customers to manage automated traffic from the same interface.
In terms of content usage control, Cloudflare has added three levels for bot management customers to define how bots use content after crawling: immediate (no storage or reuse), reference (indexing, summarization, and backlinking), and full (summarization or reproduction). The company is extending the Content Signals format in robots.txt by adding a new use parameter to express these preferences. While robots.txt does not enforce this setting, Cloudflare will report through BotBase whether verified bots comply with the stated preferences. Verified bots that ignore these preferences or fully reproduce content may lose their verified status. Previously, all verified bots were allowed by default. Under the new model, verification only confirms bot identity, and access rights depend on its classification and the website owner's policy, including whether search, proxy, or training crawlers are allowed. Cloudflare also plans to make the verification process more transparent and is developing tools to help bot operators manage their classification and verified status.
For AI agents and bots operating through third-party platforms, Cloudflare proposes a transitive trust model. The company plans to use the standard HTTP Forwarded header to identify the original requester behind intermediaries; this header can also include the operator's declared content usage method, such as use=reference. This will allow website owners to apply trust and access policies based on the original bot operator rather than the intermediary handling the request. The company acknowledges that this approach may not be suitable for all use cases where privacy or anonymous access is important.
