BigQuery

Simplify cross-engine analytics with one Iceberg table, zero copies

One Apache Iceberg table.

3 min readKDnuggets
Simplify cross-engine analytics with one Iceberg table, zero copies

The industry has spent years building bridges between data engines, only to leave the most common workflow, running analytics across multiple tools on the same data, strangled by copies and sync jobs. Google Cloud's new Lakehouse runtime catalog, which lets teams run BigQuery, Apache Spark, and Trino on a single Apache Iceberg table with zero copies, is the kind of practical breakthrough that makes you wonder why it took so long. This isn't about another storage format or a faster connector. It's about removing the friction that silently erodes productivity: the hours spent reconciling datasets, the brittle pipelines that break when an engine updates, the cognitive overhead of remembering which copy is fresh. For organizations already invested in Google Cloud, this changes the default answer to a question every data team asks: "Do we have to move the data?"

The architecture is deceptively simple. Iceberg handles concurrent commits between engines natively, so Spark can write a new partition while Trino reads the table and BigQuery runs an aggregation. No locks, no queues, no manual coordination. This directly addresses the tension we explored in our piece on Translating Data Workflows: The Missing Business Layer in Analytics Engineering, where the disconnect between technical and business logic often stems from data that lives in silos. When analysts and engineers share a single source of truth without orchestration overhead, the conversation shifts from "which copy is correct" to "what should we build next." That shift is the real value here. It's also worth noting the contrast with the approach taken by the DuckDB team in their DuckLake project, which we covered in Discover DuckLake: Build a Lakehouse on DuckDB in Under an Hour. DuckLake stores table metadata in a SQL database, offering a lightweight alternative for smaller deployments. Google Cloud's catalog is the opposite end of the spectrum: it assumes you want the full weight of Iceberg's ecosystem and provides the runtime to make it work across engines at scale. Both are valid, but they serve different users, and the market is better for having both options.

The practical takeaway is direct: if your team currently runs even two different query engines against the same data, you are likely spending more time on data movement than you think. The Lakehouse runtime catalog eliminates that cost entirely for BigQuery, Spark, and Trino workloads on Google Cloud. The open question is how this extends beyond Google's ecosystem. Iceberg is an open standard, and the concurrency model works regardless of the catalog implementation, but the runtime catalog itself is a Google Cloud feature. Teams on AWS or Azure will need their own equivalent. That doesn't diminish what Google has built; it simply sets a benchmark for the rest of the cloud providers. Watch for how quickly they respond, because this is the kind of capability that, once adopted, makes the old way of working feel like a tax you no longer have to pay.

From KDnuggets

Run BigQuery, Apache Spark, and Trino on one Apache Iceberg table in Google Cloud's Lakehouse runtime catalog, with no copies or sync jobs, and see how Iceberg handles concurrent commits between engines.

Read the original at KDnuggets

Simplify cross-engine analytics with one Iceberg table,