en.Wedoany.com Reported - Meta recently disclosed the detailed design of its storage infrastructure, stating that its system helps researchers reduce the time required to move AI training data across regions by up to 97%. The tech giant is advancing its cloud computing business unit and is expected to join the ranks of hyperscalers. In a public statement, Meta noted that graphics processing unit computing roughly triples every two years, but storage speed has not kept pace. To bridge this gap and serve storage clusters covering Meta AI, Reality Labs, social media platforms, and future cloud customers, engineers rebuilt its BLOB storage layer, which sits atop Tectonic, a multi-tenant storage structure spanning global data centers with exabyte-scale capacity.
Meta's traditional BLOB architecture was resource-inefficient for supporting AI training. Software engineers Sidharth Bajaj and Venkatraghavan Srinivasan stated that the old system stacked too many service layers and had to perform sequential stateful metadata lookups across multiple layers before resolving file paths to actual storage locations, increasing latency and reducing performance. The rebuilt storage foundation revolves around three core changes: first, consolidating the fragmented metadata system into a unified model supported by a database called ZippyDB, enabling near-instantaneous path lookups; second, eliminating the "data plane proxy" and adopting a "fat client" architecture to stream bytes directly from storage servers to clients, reducing latency while improving energy efficiency; third, shifting from a global deployment model to a regional deployment model, co-locating storage with the GPUs that actually need it.
In addition to comprehensively improving the overall storage stack, Meta engineers also utilized idle GPU memory as a distributed data cache for frequently and concurrently accessed data. The team reused experience from Owl, a system for distributing large data objects, integrating its peer-to-peer sharing logic into the BLOB storage client software development kit, so that all data accesses go through this cache, reducing the frequency of GPU requests to storage. Furthermore, Meta introduced an independent "read plan" metadata cache that returns the storage address of frequently requested files within one to two milliseconds. This combination absorbs demand spikes, such as when GPUs restart simultaneously and request the same "hot" model weights, while also improving latency.
The final revision also fixed congestion and timeout issues caused by export spikes. Meta implemented a dynamic concurrency control software solution that adjusts parallelism based on application-level congestion signals, automatically reducing the number of requests an application can send when traffic is high. According to Bajaj and Srinivasan, the revised BLOB storage stack can now support AI workloads without stalling GPUs, saving costs while providing performance advantages.
To address data delivery issues, engineers built a tiered caching system that mimics how a computer CPU pulls data from disk into faster storage hierarchies. Memory and flash on the GPU host serve as the fastest tier, regional flash BLOB storage as the next tier, with required data pulled into faster storage in advance via a "prefetch" mechanism, and HDD-based global BLOB storage as the ultimate source of truth. This new data loading paradigm has been applied to production workloads, reducing average ingestion time from 150 minutes to 10 minutes, a 93% decrease. At maximum, Meta observed that ingestion times previously requiring 89 hours were reduced to 182 minutes, a 97% decrease. Meta stated: "Modern AI workloads are data-intensive, and storage bottlenecks directly impact GPU utilization and computing costs. By rebuilding the metadata subsystem and adopting a tiered caching architecture with prefetching/on-demand hydration, we can effectively meet the demands of today's workloads."
