en.Wedoany.com Reported - Databricks has announced a technology called LTAP (lake transactional/analytical processing), with the marketing slogan "one copy of data, zero compromise, zero copy," aiming to solve the challenge of OLTP and OLAP coexisting on the same system. The technology is built on the open-source unified analytics engine Apache Spark and relies on two new products: Reyden (a new compute engine) and Lakebase (a serverless PostgreSQL based on open object storage). Databricks seeks to unify transactional, analytical, streaming, and operational data on a single storage copy within a data lakehouse (a combination of data lake and data warehouse), addressing the fundamental challenge of OLTP performing small row-oriented reads and frequent writes, while OLAP performs large column-oriented reads and batch writes. This issue has become particularly urgent as the database market pursues workloads driven by AI agent deployments.

However, controversy surrounds its "zero copy" claim. Some data engineers have pointed out that in LTAP, current PostgreSQL data is stored locally in pageserver format, then propagated to object storage in Parquet file format for long-term persistence. PostgreSQL/Lakebase can retrieve data from object storage and convert Parquet data back to pageserver format. A commentator from a Databricks competitor noted that this is effectively "two copies of data, not one." Slides presented by Databricks engineers Hristo Stoyanov and Jonathan Katz at a PostgreSQL conference in May also confirmed this, showing that pageserver provides storage while Spark analytics executors pull layer files containing full page images from a mirror layer in object storage. In a private message community seen by The Register, a Databricks engineer responded that technically there are two copies, as pageserver acts as a cache or materialization layer in the Neon architecture. Lakebase is based on technology from Neon, which Databricks acquired last year, offering copy-on-write branching and auto-scaling serverless computing.

Unifying OLTP and OLAP is not a first for Databricks. In 2014, SingleStore began researching in-memory row storage and disk-based column storage, adopting tiered storage technology that allows analytics and transaction processing on a single system. A cloud database service launched by the company in 2020 reportedly automatically manages data across three tiers: memory, local cache, and object storage. In response to Databricks' claim that hybrid transactional/analytical processing (HTAP) has failed, SingleStore CTO Nadeem Asghar responded that unifying OLTP and OLAP is precisely the goal of HTAP, and "renaming it LTAP just changes the marketing. It doesn't change the physical reality." He noted that Databricks' system runs three engines, each with its own cache and judgment on data freshness, and the physical form of "two copies of data" requires some mechanism to keep them synchronized. Additionally, MongoDB offers column store indexes, Oracle's HeatWave for MySQL allows analytics to run on transactional applications, and SAP has promoted real-time analytics since 2011 based on its in-memory database HANA.

Databricks insists its "zero copy" claim is genuine because it avoids having two authoritative copies of data that need synchronization. In a statement to The Register, a Databricks spokesperson said that in LTAP, users operate on a single authoritative copy of data, which has a single source of truth in Iceberg, while other intermediate internal copies belong to the "database storage hierarchy." At a recent conference, Databricks qualified this by stating that data has only one "authoritative" copy or only one copy "in storage," similar to SingleStore's description of its storage layer as "transparent to users."

Andy Pavlo, an associate professor of databases at Carnegie Mellon University, noted that regardless of marketing debates, Databricks has accomplished impressive engineering work in enabling the new engine Reyden to read PostgreSQL pages. He stated that Reyden's ability to interpret PostgreSQL page content is no small feat, as pages are not entirely self-contained and require returning to Neon/PostgreSQL to retrieve metadata from the catalog. The technology allows for more timely analytics, performed in a transaction-safe manner without waiting for data to be pushed to S3. Meanwhile, Reyden is stateless and can scale horizontally by adding compute resources. Pavlo also acknowledged that "they do end up copying data."