en.Wedoany.com Reported - On July 8, U.S. cloud services and network infrastructure company Cloudflare and U.S. artificial intelligence company OpenAI launched a research pilot project to explore how insights from participating websites on Cloudflare's global network can help AI search engines more efficiently discover, crawl, and index open web content. The project focuses on the accuracy and timeliness of AI answers, with core signals including content freshness, traffic quality, and actual page changes.
The biggest difference between AI search and traditional search is that results are no longer presented solely as web page links but are directly generated by the model. Whether a page has been updated, whether its content has actually changed, whether traffic is abnormal, and whether the site's content remains valuable for citation all affect the AI system's ability to obtain more accurate web page information in a timely manner. Cloudflare is not providing the actual web page content itself, but rather real-time signals from the network and site layers to help the AI system determine which pages are more worth crawling, when to recrawl, and how to reduce ineffective indexing.
Cloudflare's global network covers over 20% of websites, enabling it to observe large-scale web page updates and access behavior. OpenAI provides the model, search, and question-answering systems, using real user queries to test the crawling and indexing effects driven by these signals.
The technical direction of the pilot project is "signal-driven crawling." Traditional crawlers typically crawl web pages based on fixed frequency, sitemaps, link discovery, or historical weight, which can lead to two types of problems: one is that pages have been updated but the AI index has not been refreshed in time, resulting in outdated answers; the other is that pages without substantive changes are repeatedly crawled, wasting crawling resources and increasing website access pressure. By introducing Cloudflare's real-time network signals, the AI search system can adjust crawling priorities based on content changes, access quality, and page status.
Content freshness is one key parameter. News, announcements, product pages, pricing pages, policy documents, technical documentation, and blog posts change at different frequencies. If AI search cannot determine whether a page has truly been updated, it may repeatedly process old content or miss new information. Actual page change signals help the system identify content-level changes, rather than relying solely on URL existence, timestamp changes, or site submissions to decide whether re-indexing is needed.
Traffic quality is more about assessing web page credibility and access status. Abnormal traffic, low-quality access, invalid requests, bot visits, and temporary page fluctuations can interfere with the AI system's judgment of a page's value. By introducing network-side signals into the pilot, Cloudflare allows OpenAI to test whether these signals help improve page filtering, crawling rhythm, and answer generation quality.
The project is also related to AI crawler governance. Cloudflare has previously launched tools for website owners related to AI crawler control, content signals, and access permissions, allowing websites to more granularly express preferences for search indexing, AI input, and model training. This pilot with OpenAI focuses more on "how discoverable content can more efficiently enter AI search," rather than having the AI system indiscriminately expand its crawling scale.
The project is currently in the research pilot phase, with the main goal of verifying whether real-time network signals can improve AI search indexing efficiency. Testing will focus on areas such as crawling priority, page change identification, index refresh speed, reduction of low-value crawling, answer timeliness, and open web content discovery capabilities. Cloudflare and OpenAI will continue testing around the scope of participating websites, signal usage boundaries, content permissions, site control, and AI search result quality.
