Change Data Capture (CDC) has become one of the foundational patterns of modern data systems. At a high level, it’s a simple idea: instead of periodically copying entire datasets, we track changes as they happen. Databases emit inserts, updates, and deletes as a continuous stream of events, and downstream systems replay those events to reconstruct the latest state. In practice, this pattern fits naturally with streaming systems like Kafka, which act as an ordered log of changes, enabling real-time processing and distribution.

As architectures have evolved toward lakehouses, however, CDC’s reach has expanded. Apache Iceberg, for example, plays a very different role from the database copies that were the initial remit of CDC but the concept of a continuously updated data lake is too attractive to pass up. To emphasize the point, Iceberg is currently doing to disparate and fragile ETL interfaces what Kafka did to operational interfaces a decade ago. Having CDC data in Iceberg makes it available to an entire organisation, not just an isolated database endpoint.

Iceberg is designed as an analytical storage layer, however, optimized for large-scale reads over immutable files in object storage. The traditional pipeline capturing CDC from a database, streaming it through Kafka, and materialising it into Iceberg quickly encounters issues that expose limitations in Iceberg’s design for fast data.

The root of the problem lies in a mismatch of granularity. CDC operates at the level of individual rows, capturing each change as a discrete event. Iceberg, by contrast, operates at the level of files. Once a file is written, it is effectively immutable. This means that even the smallest update cannot be applied in place. Instead, the system must locate the file containing the affected row, mark the original version as deleted, write a new version of the row into a new file, and then commit an updated snapshot of the table. What appears to be a trivial change at the logical level becomes a surprisingly heavy operation at the storage layer.

Over time, this mismatch creates compounding issues. Small updates trigger disproportionately large amounts of work, leading to a proliferation of small files and metadata. Deletes and updates are not spared, these accumulate as separate delete files, which increase the cost of query planning. None of these problems are insurmountable, Iceberg has compaction and deduplication processes able to restore efficient data and metadata layouts but these are heavyweight and their execution must also be carefully managed. In short, the operational overhead is large enough to consider whether the result is really worth it in the end?

Streambased’s approach starts by embracing a distinction between log based event stream and materialised tables instead of fighting it. Rather than insisting that all data must be materialised into Iceberg before it can be queried, the dataset is split into two complementary parts. One part (the “coldset”) is fully materialised and lives in Iceberg, representing the stable, finalised, historical view of the data. The other part (the “hotset”) remains in Kafka as an ephemeral stream of recent changes. Queries operate across both, combining the durable cold data with the live hot data to produce a complete and up-to-date view. The sections are composable with the size of the two sections balanced according to the workload they serve. An easy to explain (but very naive) composition may expand the hotset (say last 6 hrs) for workloads that expect many updates and deletes to avoid accruing metadata in Iceberg but may have a much smaller hotset (say last 15 mins) for workloads that are primarily insert of new rows (a much less expensive operation in well partitioned Iceberg).

This shift has a subtle but powerful effect. By allowing recent data to remain in Kafka, the system avoids the constant pressure to translate every small change into file-level operations. In this sense, materialisation becomes a controlled, deliberate step rather than a continuous background process. Operators are free to make informed decisions about how and when materialisation happens and can accept the right tradeoffs for their use cases.

This model also aligns closely with how data behaves in real systems. Most datasets exhibit a clear distinction between recent, high-churn data and older, relatively static data. The last few hours or days tend to be latency-sensitive and subject to frequent updates, while older data is rarely modified and is primarily accessed for analysis. Kafka is well suited to handling the former, providing low-latency access to recent events, while Iceberg excels at storing the latter efficiently and cheaply over long periods.

By splitting the dataset along this natural boundary, the system can take advantage of both technologies without forcing either to operate outside its strengths. Kafka retains its role as the system of record for recent changes, enabling real-time querying and processing, while Iceberg provides a scalable, cost-effective store for long-term data. Importantly, this also reduces the need for heavy operational machinery. Many of the maintenance steps that plague traditional CDC pipelines: compaction, snapshot cleanup, repartitioning, become either less frequent or unnecessary, because the system avoids generating excessive intermediate state in the first place.

Another consequence of this approach is that it restores the immediacy that CDC promises. In a fully materialised pipeline, there is always a delay between when an event occurs and when it becomes visible in Iceberg. Queries are effectively bounded by the last successful write. In contrast, when queries incorporate the live Kafka stream, newly arrived events are immediately reflected in results. The system is no longer querying a slightly outdated snapshot but is instead computing state directly from the most recent data available. The cost of this “zero latency” is a small amount of extra work at query time, where the system reconciles the Iceberg snapshot with the latest events in Kafka. Instead of reading a fully precomputed table, the query “plays forward” the recent portion of the log to produce the current state.

In practice, this is a favourable trade. The additional compute is limited to the hot data window, while the system avoids the much larger ongoing costs of constant writes, compaction, and maintenance in Iceberg. Rather than paying continuously for freshness, you pay only when you query, and only over a small slice of data.

Ultimately, this leads to a different way of thinking about CDC in the lakehouse. Instead of viewing it as a process of continuously writing changes into tables, it becomes a matter of composing views over a combination of log and storage. Kafka holds the evolving, mutable edge of the dataset, while Iceberg holds the stable, immutable core. The full state emerges from the interaction between the two, rather than being fully materialised in either system at all times.

The real issue isn’t that CDC is hard in Iceberg, it’s that we’re solving the wrong problem.

Fully materialising every change is an assumption carried over from older architectures. In a log-first world, it’s unnecessary. The log already contains the truth; Iceberg only needs to store the parts of that truth that have stabilised.

Once you accept that, the solution becomes obvious: stop writing everything, all the time. Let Kafka handle change, let Iceberg handle history, and draw from both as required and only when you need to.

3 months ago I posted: The 9 Ways to Move Data from Kafka to Iceberg, an in depth look at a growing ecosystem of tools and architectural patterns for landing streaming data into an Iceberg lakehouse. I mapped out everything from “classic” connectors to newer table-native and managed approaches. If you haven’t already, check it out here:

But data systems rarely flow in only one direction. Once Iceberg becomes the system of record, the next question is how to get that data back out, into Kafka, so that downstream services can react to failures, rehydrate and counter data corruption. All of these scenarios require the reintroduction of lakehouse data into the event-driven world (and beyond). In this follow-up, I’ll walk through three practical ways to move data from Iceberg to Kafka: building it as a connector, leaning on managed services, or using complementary services that sit on top of an existing Kafka/Iceberg estate.

I’ll focus on 3 main reasons for inverting the typical Kafka -> Lakehouse flow:

1. To bootstrap new workloads - When a new Kafka workload comes online it often needs history before it can do anything useful.You can’t compute aggregates, build search indexes or warm caches with data starting from right now. What’s more, Iceberg retention is generally much longer than Kafka, giving a far richer context to draw from.

2. To recover from outages/poison pills - Production Kafka streams fail in messy ways: bad messages, schema breaks, consumer bugs, downstream outages. When this happens, teams usually need to “rewind” and replay from the point of failure to recover. The tricky part is that poison messages can go undetected for days/weeks/months so the amount of data you need to replay can be huge. Iceberg is ideal here: it retains large volumes cheaply, has powerful updates to fix bad data, and makes it practical to re-emit a correct stream without needing Kafka to hold months of retention.

3. To save costs in exceptional circumstances - Kafka retention is expensive at scale. If a consumer falls behind (deploy issues, downstream throttling, sudden traffic spikes), the default solution is to keep enough Kafka retention to cover the worst case. But that means paying for expensive “just in case” storage 24/7 and worse, because the data is transferred “ahead of time”, duplicating your real-time dataset in your lake. A cheaper pattern is to let Kafka stay optimized for real-time retention, and use Iceberg as the long-term buffer.

Section 1: Using a Connector

At the time of writing, there isn’t a true “Iceberg source connector” you can just drop into Kafka Connect and call it a day.

But you can still build a connector-based Iceberg → Kafka pipeline today by leaning on your favourite Iceberg query engine. For example: run Trino on top of Iceberg, then use the Kafka Connect JDBC Source Connector to query Trino and publish results into Kafka.

At that point, the pattern looks like any other database connector:

1. Poll on an interval

2. Run a query

3. Publish the resulting rows into Kafka topics

The main upside of this approach is that it’s straightforward and uses well understood, battle-hardened components: Kafka Connect plus a JDBC source connector is a pattern most teams already know how to run, scale, and observe, and it lets you reuse whichever Iceberg query engine you already have (Trino, Spark, Dremio, etc.).

The tradeoff is that there’s no end-to-end linkage between the Kafka events that originally landed in Iceberg and the records you’re now emitting back into Kafka. That lack of continuity makes it a poor fit for “Iceberg as Kafka retention” use cases like backfill and cost-saving replay, where you need stronger guarantees about exactly what range of data is being re-emitted.

Section 2: Kafka Compatible

Kafka compatible services preserve the Kafka client experience (protocol, offsets, consumer groups) but use different engines to provide the underlying storage and metadata semantics. In the world of Iceberg this usually means the storage layer is significantly different to the custom log format and disk based approach used by Apache Kafka.

StreamNative Ursa

Ursa enables Kafka clients to work directly with Apache Iceberg data by replacing the traditional Kafka broker model with a compatible streaming engine that writes all streamed events as part of a unified lakehouse dataset that is eventually stored in open table formats like Iceberg. Ursa’s engine first ingests data into a streaming focused layer before persisting it to a durable Iceberg layer.

Ursa adds a Kafka-compatible read path that maps Kafka’s offset-based consumption model onto Iceberg’s file-based table layout. When a Kafka consumer issues fetch requests, Ursa resolves the requested offset range into the relevant Iceberg snapshot and then plans reads over the underlying Iceberg files, returning records through the standard Kafka protocol as if they came from a normal Kafka log.

This approach has a much cleaner operational model than the connector approach (StreamNative takes care of stitching real-time and analytical datasets together) and lends itself to a more cloud native deployment (StreamNative offers Ursa in their cloud product suite).

Ursa combines streaming log storage and Iceberg, translating Kafka offset-based fetches into planned reads over Iceberg table snapshots/files, before returning results through the Kafka protocol as if Iceberg were a regular collection of topics.

Unfortunately Ursa brings with it all the usual issues with compatible systems, at the end of the day your underlying data layer is not really Kafka and so is subject to feature drift and unexpected differences in behaviour (for instance Transactions and topic compaction are not fully supported yet). Additionally there are vendor lock in concerns, the Iceberg -> Kafka path may not be relevant for all workloads and yet, with Ursa, the entire streaming stack is locked in.

Bufstream

Bufstream’s Kafka compatible engine writes Kafka topic data directly into Parquet files and Iceberg metadata, skipping the intermediate streaming layer from StreamNative’s approach. It then uses those same Parquet files to serve Kafka consumers or Iceberg clients.

The advantage of this approach is its simplicity. With only a single storage layer it is easy to ensure consistency of the two views and use cloud native storage options for cost reduction. Unfortunately this comes with a cost in write latency, object storage is on the write path and adds producer latency. Bufstream typically experiences 3-5x higher end to end latency (P99) than traditional Kafka.

Bufstream takes a simple “single storage layer” approach by writing Kafka topic data directly into Iceberg and serving both Kafka consumers and Iceberg readers from the same files. Having one single storage layer means that the Iceberg to Kafka flow is a core requirement of the service. In Bufstream both Kafka readers and writers interact with Iceberg data translating formats on the fly as required.

Section 3: Complimentary

This category covers systems that augment an existing Kafka + Iceberg estate rather than replacing either layer. These solutions typically sit alongside Kafka brokers and Iceberg tables as add-ons. The main benefit is they don’t require any migration and do not lock in a user’s whole streaming stack.

Aiven Iceberg Topics

Aiven’s Iceberg Topics concept sits on top of an existing Apache Kafka deployments and leverages a custom Tiered Storage Manager (RSM) plugin to read/write data to Iceberg. As Kafka log segments roll, their data is transferred to Iceberg and tracked inside the Kafka cluster. On fetch the broker can reconstruct valid Kafka batches from those same files, effectively allowing Kafka consumers to replay data that lives in Iceberg storage without a separate layer or copying process.

The semantics of all Kafka client operations are preserved because the core idea is “just Kafka” but you gain an analytical view on the data that can be consumed by Iceberg engines. This approach elegantly leverages existing Kafka tiered storage functionality but, because of this, requires back end control of the cluster and for the cluster to be running the Apache Kafka flavour. For this reason Iceberg topics cannot be used with managed Kafka deployments or “Kafka compatible” systems. On top of this the github project that backs this feature appears stale and very much for testing only.

Streambased

Streambased functions as an abstraction layer above pre-existing Kafka and Iceberg deployments. Streambased composes a dataset made up of real-time focused data from Kafka and analytically prepared data from Iceberg and serves this to either Kafka clients or Iceberg clients in the format they are expecting.

Streambased exposes 3 datasets in this way:

1. The hotset - data that resides in Kafka alone

2. The coldset - data that resides in Iceberg alone

3. The mergedset - a seamless joining of the hotset and coldset

By exposing these 3 datasets as Iceberg, users can easily move data between hotset and coldset, structuring the mergedset in a way that is optimal for the workloads it must enable.

Under the hood, Streambased serves Kafka data in a similar manner to StreamNative’s Ursa above. Streambased intercepts fetch requests, computes the location of the relevant data (in Kafka, Iceberg or a combination of the two) and serves it back via the Kafka protocols. Where Streambased differs is that it is agnostic to the underlying data systems. Ursa requires that data be written to it before it is accessible whereas Streambased plugs into any system that implements the Kafka protocol (Apache Kafka, managed providers and compatible systems).

This makes Streambased a complementary layer that upgrades an existing Kafka + Iceberg estate with unified access and consistent semantics, rather than replacing core infrastructure.

Conclusion

Iceberg is rapidly becoming the durable system of record for streaming data. However, the moment you treat it that way, you also need a clean path back into Kafka. With this established, applications can replay history, recover safely, and keep real-time systems lean.

The Iceberg → Kafka pattern you will choose depends on the existing environment and workload constraints. Connectors offer a pragmatic starting point, Kafka-compatible engines rethink the storage model entirely, and complementary approaches unlock replay and unified access without replacing your existing stack. The right choice depends on what you’re optimizing for but the direction is clear: the lakehouse isn’t just the destination anymore, it’s becoming part of the streaming runtime.

Subscribe to follow how Iceberg and Kafka are converging and what that means for your architecture.

Subscribe now

Over a decade ago Martin Kleppmann predicted we’d turn the database inside out and that Kafka would be in the middle of it.

https://martin.kleppmann.com/2015/03/04/turning-the-database-inside-out.html

Modern enterprises that embraced event streaming have essentially turned their monolithic databases inside out into distributed components that resemble a database when looked at holistically. Martin was right but Kafka is not the only star. Open table formats are the turning point where we finally turned the database inside out - they allow organizations to build their own Snowflake.

Martin talked about 4 things:

• Log Replication

• Materialized Views

• Secondary Indexes

• Caching

Of these four, two of them (Replication and Materialized Views), have successfully transitioned from internal database implementation to external core components of stream processing:

• Kafka and the rise of event streaming has featured ingestion of events into an externally, durably-replicated facing log

• Materialisation of these events in a downstream view (e.g. state in Streams/Flink or DB tables via Kafka Connect).

But what about the other two? It’s been rare to hear about secondary indexes or caching in our day to day life as event streaming developers. These remain opaque, duplicated and fragmented inside the downstream systems fed by event streams.

For instance, it was not until you materialized a stream into e.g. Postgres that you could apply indexes and perform fast point lookups on it.

Streaming pipelines generate immense volumes of immutable data, but ad-hoc querying or joining across that data efficiently remains practically impossible due to a lack of secondary indexes.

Similarly, from the beginning caching was crucial for performance when data was repeatedly accessed and transformed. Kafka had basic caching inside it but nothing for external systems which required format conversion. For example, in retail cases transactions data was landed into Kafka. This data was then transformed and loaded into Logistics (for stock levels) and Marketing (for recommendations) domains via Connect. A cache avoided the two flows processing the same data in the same way twice (why burn that extra cpu?).

What’s more, turning the database inside out didn’t stop with Kafka. With the advent of open table formats such as Apache Iceberg more and more of data storage and processing internals were exposed. Iceberg exposed previously internal concepts like partitioning, versioning, logical deletion and statistics to allow processing engines to work with data at previously impossible levels of flexibility and performance.

In this article I’m going to explore these database concepts (and more) through the lens of Kafka and Iceberg to challenge how we think about the boundary between streaming systems and data lakes and how Iceberg might just be the bridge that finally unites them.

Part 1: Exposing the WAL again

Inside a modern database, new data is written to a Write-Ahead Log (WAL) before being persisted to long-term storage (like a diary the database keeps, noting down everything that happens so it can retrace its steps if needed). A decade ago, Martin Kleppmann championed the idea of making the WAL externally visible, streaming it through systems like Kafka, while pushing long-term storage into lakes via ETL such as Kafka Connect.

This separation unlocked real-time architectures, but it also created a new problem: Data became split into a hotset (WAL) and a coldset (data lake), and recombining them became the application’s burden.

A database owns both layers and so can seamlessly merge “just-written bytes” from the WAL with “years-old pages” on disk and present them as one coherent table

Caching, buffer pools etc. make this fast and transparent. But once we turn the database inside out, we lose this built-in functionality.

In the inverted architecture, Kafka becomes the WAL and as such is a short term buffer into more permanent storage. Kafka was never designed for long retention and keeping large retention windows is expensive and impractical.

For the long-term storage destination, the answer today is clear: Apache Iceberg

It provides:

• Cheap durable object storage

• More complex access patterns (SQL)

• Schema and partition evolution

Unfortunately, streaming ingestion into Iceberg causes unmanageable metadata explosion.

Fundamentally, the WAL (Kafka) and the lake (Iceberg) speak different languages, records vs. files, offsets vs. snapshots. Each is highly attuned to it’s specific role.

The missing piece: a layer that projects Kafka into Iceberg. We call this Streambased I.S.K. (Iceberg Service for Kafka). I.S.K. is a database-like fusion of WAL+ long term storage, that functions as a cache above Kafka that:

• Mirrors the structure of Iceberg tables, not the structure of Kafka topics

• Presents the hottest window of data remaining in Kafka + cold data already in Iceberg

• Dynamically translates Kafka records into Iceberg-shaped files/metadata

This allows Iceberg clients to see one logical table, clients query a single table but under the hood, the cache is doing the work of DB internals.

“But Kafka already caches data!”

Yes, Kafka relies heavily on the Linux page cache. This is what makes it fast for consumers: recent reads and writes are served directly from memory.

But this cache:

• Is accidental, not controlled

• Is record-oriented, not table-oriented

• Does not mirror the layout or semantics of Iceberg tables

Great for stream consumption, but not the caching layer you need to unify stream+lake data.

A true “inverted database” needs more than a visible WAL and scalable lake storage, it needs a caching layer that reassembles them into a single, coherent table the way a traditional database always has.

Part 2: Indexing

Does Kafka have indexes?

I asked this question in a recent talk at Confluent’s Current Conference and only 2 thought it did. Even engineers that have worked on Kafka for years aren’t aware of Kafka’s built in indexing.

Kafka actually maintains two indexes:

• An offset index maps message offsets to physical byte positions in the Kafka data files - this index is used to quickly find the correct position in the data to read from.

• A time index maps message timestamps to message offsets - this allows you to seek a particular time in a topic quickly and easily.

Don’t take my word for it, if you open the data directory in your Kafka cluster you will see files ending “.index” and “.timeindex” alongside the log data.

These indexes don’t physically reorder data to answer queries faster. Instead, they layer lightweight additional structures that map question (“what happened at time x”) onto the original data layout.

The same idea can be applied in Iceberg where data is physically laid out in partitions. A partition is a collection of data files that share common attributes. For instance, partitioning may group financial transactions by account no., making it faster to calculate the balance for an account.

Unfortunately, because partitioning affects the physical data layout, it’s difficult to optimise for multiple query patterns. If we wanted to search our financial data by transaction id, partitioning by account doesn’t help and the engine would need to visit every account partition to find the correct data.

We can address this problem in the same way Kafka addresses querying by timestamp: by layering indexes on top of each other to resolve the correct data. With a new index that maps transaction id -> account no.. our Iceberg engine can lookup by transaction id with the same performance boost as if it was looking up by account no. The flow is this:

1. Find the transaction id in the index and find the account no(s) associated

2. Query the table for the account no. pruning away unneeded partitions

3. Filter the (much smaller) result set for the transaction id.

Iceberg already provides the tools for this. An Iceberg table is the ideal place to store the index and applying the index is a case of adjusting the query:

SELECT *
FROM  transactions_by_account_no tab
JOIN transaction_id_to_account_no index
ON  tab.account_no = index.account_no
WHERE index.transaction_id = ‘1234’

Given the above, Iceberg engines (Spark/Trino etc.) are able to follow the join and execute the query faster.

The principle is the same, Kafka translates timestamps → offsets → bytes; Iceberg translates predicates → partitions → files, both without changing the layout.

This is not a silver bullet however, for translation layers to work well the data it is addressing should be naturally-clustered (spoiler alert: a LOT of data is naturally clustered) and high cardinality. Timestamps are a great example of this, in most cases Kafka data is roughly time ordered anyway, we can expect records to be loosely clustered by timestamp but because we cannot guarantee this, selecting a time range in Kafka without an index involves a full scan. Kafka’s time index takes advantage of this naturally clustered layout to quickly find the clusters we are interested in and save a lot of scanning.

Another great example is a sessionId. Sessions are generally short-lived, so you have high certainty that events for a given session are grouped close together. This means:

• An index on sessionId allows lookups to jump to a small contiguous region(s)

• Reads on the underlying data stay mostly sequential and cheap

• A predictable, bounded amount of data to read per session regardless of total topic size

That last point is important because it highlights why this scales: the cost of the lookup grows with the session size not with the log size.

You see the same pattern with request IDs, trace IDs, transaction IDs, common fields that already have great locality.

Not all data can leverage translation layers though. Consider something like a product category: A low cardinality field that would usually be evenly spread across the entire offset range. If we apply our translation technique here we end up with the following roadblocks:

• Every value maps to offsets across the entire topic

• Index lookups fan out into many small, non-sequential reads

• The end result is close to a full scan of the underlying data

Instead of looking closer at the data, we often reach for practices that expensively rewrite the underlying data (repartitioning, compaction etc.) or move it to entirely new storage layers. All the while ignoring the fact that the existing layout could already support fast access with a thin translation layer on top. This is what we’re building in Streambased.

Part 3: Too much indexing

In the previous section we talked about what’s possible with indexing via translation structures. However, in data streaming today, indexing typically means materialisation of events into a state store. From this separate store, streaming data is read, re-organised and stored duplicately in a layout more suitable for access by future operations. For instance, taking a stream of financial transactions and using them to construct a table that is organised by the account number the transaction relates to.

Traditionally, these materialisations are built directly into the streaming applications that use them. Applications individually load source Kafka topics and materialise stores for their use and their use only. If you have 3 applications that need the same store that’s 3 duplicate copies of the data. This is in direct contrast to database style indexing where indexes are built once and available to all. Platforms like Flink and KSql went some way to address this inefficiency by treating all streaming flows as a single distributed application but it is still not common practice to reuse the materialisations of one flow in another.

Unlike in the database (where a database engine operates over all indexes), there is no component in streaming that ensures materialised tables are consistent, not duplicated and optimized for the tasks they perform across all applications. The result of this missing component we call “index fan out”: a new pipeline, a new sink, a new copy of the same data for every materialisation and no consistency guarantees between them.

How can we address this limitation? By centralising the index. Creating a single authority for what is indexed where. All we need is a central place to store this indexing data: enter Iceberg. The ideal open and central place for applications to source from.

Fan-out indexing isn’t evil, it’s a workaround for missing primitives and a central management component. As logs and tables converge, the opportunity arises to stop duplicating data and start indexing it properly.

Part 4: Deduplication

On our path towards turning the database inside out there’s an elephant in the room that it’s finally time to confront: data duplication between streaming and analytical systems.

In a typical modern architecture, data lands in a streaming system, then gets ETL’d into an analytical store. Streaming applications read one copy, analytical applications read another and a multi billion dollar industry has grown up around transferring between the two.

However, by taking this approach we’re not just moving bytes around, we’re creating two (or many more) versions of reality. One might be a few seconds behind. One might have a slightly different schema. One might silently drop a field or re-order events. And now every downstream team has to ask the same uncomfortable question: which one is correct?

What if the problem isn’t ETL, but the boundary we’ve chosen?

Instead of treating streaming and analytics as two separate worlds that must be synchronized after the fact, imagine storage being composed of two parts: one optimized for streaming access and one optimized for analytical access. On top of that, a shared abstraction lets applications interact with the data without caring where a particular read or write is physically served from. This is the principle Streambased is founded around.

Earlier in this article I wrote about consuming WAL and long term storage elements of data architectures and it is these that create this composed dataset. In our inverted architecture Kafka provides the WAL element and Iceberg the long term storage. These are joined for analytical purposes by Streambased I.S.K. This lets analytical systems query live streaming topics as if they were just another set of tables, giving teams immediate access to the freshest data alongside full history in one place. All without ETL, duplication, or lag.

But what about streaming? This idea works in both directions, if we can compose storage for analytical workloads, why can’t we do the same for streaming workloads? That’s where Streambased K.S.I. comes in, instead of forcing Kafka clients to live only in the kafka data and leave the cold, analytical data unreachable. K.S.I. maps Iceberg’s rich historical store back into a Kafka-like interface so streaming systems can see the full timeline as a single, logically continuous topic.

Turning the database inside out again

To really “turn the database inside out” we need to stop thinking of Kafka and Iceberg as two separate destinations, and start treating them as two layers of the same logical system. Kafka is the hot edge: a fast, append-only WAL optimized for fan-out and low-latency consumption. Iceberg is the durable core: a table abstraction over cheap object storage that makes history queryable, evolvable, and governable.

The mistake the industry made was assuming the boundary between these two layers should be crossed with ETL, rather than bridged with database primitives.

Once you accept that, indexing and caching stop being “features of downstream systems” and become shared infrastructure again. Meaning:

• A translation layer can project the log into the table, preserve locality, and expose secondary indexes as first-class assets rather than bespoke application code.

• Materializations can be built once and reused everywhere, with consistency guarantees that feel more like a database than a pipeline zoo.

• A single source of truth dataset can be consumed by operational and analytical clients in the formats they expect

In that world, the question isn’t whether Kafka replaces the database or Iceberg replaces the warehouse. The real shift is that the database becomes a set of open, interoperable components (log, cache, indexes, and tables) and we get to assemble them without giving up correctness or performance. Kafka was the start of the inversion, adding Iceberg makes it permanent. And the missing bridge between them is where the next decade of data infrastructure will be built.

Apache Iceberg is the latest entry in the never ending quest for a simple, single way to store and process data at scale. Taking over from the enormously successful Apache Hive, it was created to:

1. Provide a single store of large amounts of raw data in a very cost effective way - in the modern cloud native world this means object storage.

2. Allow consumption of this data using a universal language (SQL)

Iceberg was initially built to solve Hive’s problems:

1. Slow query planning at scale (Hive required listing every partition)

2. Lack of ACID guarantees (concurrent writes could corrupt tables)

3. Schema evolution required full table rewrites

4. No time travel or snapshot isolation

And perhaps more importantly:

1. Provide an open, well-documented standard. in data serving that can be implemented in many engines (no vendor lock in).

2. Scale to handle the largest possible datasets (essentially infinite)

As Iceberg adoption began growing people quickly saw the opportunity afforded and began looking beyond simple batch workloads to more modern data ingestion approaches. As usual Postgres is ahead of the game, the team at Mooncake Labs built Moonlink: a real-time ingestion engine that moves data from Postgres directly into the open-table format Apache Iceberg. This was recently acquired by Databricks and similarly, Snowflake’s acquisition of CrunchyData produced pg_lake with a similar goal of unifying Postgres and downstream Iceberg.

Data streaming needs to catch up! Unfortunately, however, real-time Kafka workloads such as these are not a good fit for Iceberg where every new write must:

1. Generate data files (e.g. Parquet files)

2. Write manifest files describing those data files

3. Atomically update table metadata

Today an intermediate system is needed to ingest Kafka data and provide an analytical interface to it. Many powerful options exist (Clickhouse, Pinot, Druid, etc.) but they all lack the “single store” benefits offered by a lake approach. If you wish to query both real-time and batch data (from Iceberg) today, complex extra steps (e.g. Confluent Flink snapshot queries) are required to combine the two views of the world.

What does great look like?

The above is far from optimal but what would great look like? The answer to this is simple. The optimal solution would provide a seamless dataset that stretches all the way from the newest real-time data point (written 1 nanosecond ago) to the oldest point available (written 5 yrs ago). Users would be able to interact with this view using the tools and techniques they use today (SQL) without adopting any special procedure or additional technologies. In fact, the surest sign that this optimal solution has been achieved is that its users do not even know it exists.

What if everything was Iceberg? What if everything was Kafka?

Achieving the above is actually remarkably simple, we just take the interfaces offered by each system and apply them globally. We need something akin to a proxy layer that serves Kafka data as if it was Iceberg and Iceberg as if it was Kafka. Users could then use the proxy layer to fetch unified data, or skip it and access the underlying system directly when their requirements are satisfied.

Because the proxy layer would need to understand both Kafka and Iceberg, it can also trivially automate the movement of real-time data to the lake. This is exactly what we created at Streambased!

The How…

First some ground rules:

1. Building a seamless real-time/batch view must not compromise the performance of either system in isolation

2. The solution should not involve data movement (No ETL), the real-time and batch systems should always remain the source of truth for their data

To achieve this, we have two products - I.S.K. (Iceberg API on Kafka) and K.S.I. (Kafka API on Iceberg).

One Table Across All Time

Streambased I.S.K. presents a set of Iceberg tables in which the underlying storage can be composed of a section of real-time data from Kafka (the hotset) and a section of physical Iceberg data (the coldset). Tables in I.S.K. combine these two datasets in a way that is completely transparent to any clients interacting with it (it just looks like a regular Iceberg table).

For example, with I.S.K., ’SELECT * FROM transactions’ executed from Snowflake would retrieve records stored in both the transactions Iceberg table and the transactions Kafka topic and mix them together to provide a seamless table.

The I.S.K. architecture consists of the following components:

1. A storage gateway - Iceberg is expecting files so I.S.K. must have a way to provide a file based interface to engines. I.S.K. presents an Amazon S3 compatible API to engines that can serve both metadata and data files with data sourced from Kafka.

2. An Iceberg catalog - I.S.K. presents a simple, read only, catalog for Kafka data, this is the entrypoint for Iceberg engines.

3. A cache - To reduce impact on the Kafka cluster and improve Iceberg performance, I.S.K. caches files served by the storage gateway. These files represent sections of immutable Kafka log and so can be cached and invalidated at will.

4. An indexing engine - Most Iceberg queries will not address the entire dataset. The Kafka API does not allow access patterns that easily address subsets of data. To address this I.S.K. maintains indexes that map Iceberg partitions -> Kafka offsets, making Iceberg engines able to prune away the Kafka data they do not need.

One Stream Across All Time

Streambased K.S.I. presents a number of Kafka topics in which the underlying storage can be composed of the same “hotset” section of data served directly from Kafka and “coldset” section served from Iceberg that I.S.K. uses above. K.S.I. maps columns from the Iceberg tables to Kafka’s partition and offset concepts allowing Kafka clients to interact with them as if they were Kafka topics.

For example, with I.S.K., a console consumer execution like:

kafka-avro-console-consumer --topic transactions --bootstrap-server ksi:9192 --from-beginning

Would start by reading older records from the transactions Iceberg table before progressing to newer records in the transactions Kafka topic.

The K.S.I. architecture is simpler than I.S.K. as the access patterns are a lot more limited. It consists of:

1. An Iceberg engine - to fetch table formatted data

2. A row processor - to converts Iceberg table rows into Kafka messages - This component reformats the column oriented Iceberg data into the key/value based messages Kafka clients expect. Governance steps like Schema Registry integration are applied here too.

3. A proxy (we use the open source Kroxylicious) - to serve Kafka clients. Most requests/responses will be passed through to the underlying Kafka cluster but fetch requests that reference cold stored Iceberg data will be served by K.S.I. and not the underlying cluster.

With I.S.K. and K.S.I., users can query and consume data continuously from the freshest event to the oldest record without ever worrying about synchronization or data duplication.

TL;DR: Streambased makes real-time and historical data behave like a single Iceberg table or a single Kafka topic, eliminating ETL, preserving performance, and unifying streaming + batch workloads on one logical layer.

Stop stitching systems together. Start using a real-time lakehouse that just works.

If you’d like to read about what we learned building this:

Subscribe now

If you were to intentionally design two protocols/formats with poor interopability, you would be hard pushed to create something worse than Kafka protocol and Apache Iceberg. Both standards were developed and optimised for entirely different purposes and share almost nothing in common:

That said, the promise that the unification of these two projects brings is too tempting to pass up. If we were able to combine Iceberg and Kafka, we’d have a single continuous view of data from right this very millisecond all the way back to the beginning of time. No data inconsistency, no mental overhead in figuring out where to pull data from.

For this reason almost all vendors are currently building towards this goal. In flows of this type Kafka is the entry point, providing an easy place to land data for real-time serving to the more latency sensitive use cases. Things like microservices, web apps and observability platforms that need to give you answers in seconds, not hours.. Iceberg, on the other hand, provides long term storage for tasks that can wait for hours or days. Things like reporting, strategic decision making and AI/ML tasks.

Most importantly, any single data point has its place in both systems. I t may be processed in your microservice for your webapp, and then recorded for longer-term analysis by your CEO. This creates a natural flow of data from Kafka to Iceberg but this flow is not without its twists and turns. Let’s look at the 4 big issues:

Issue 1. Data Freshness 🕒

Kafka likes to pass messages around in small batches (~16KB). Conversely, Apache Iceberg works best when data is written in large chunks (~512MB). This mismatch is usually addressed by adding a step in the transfer flow that accumulates a number of small messages (usually driven by a total size in bytes or a time period) into a larger chunk before writing to Iceberg.

This inevitably results in time spent waiting. The downside of that is a reduced freshness of data available in Iceberg. The accumulation period means no Iceberg ingestion is happening until the batch size is hit. This creates a minimum lag time (> 15mins). In practice, this means you’re never able to use your beloved SQL in describing something that happened now

Issue 2. Table Maintenance 🧹

Apache Iceberg requires that a number of recurring maintenance procedures are executed on any dataset stored within it. The most common of these are:

1. Compaction - Iceberg is a file based format and, as mentioned above, Iceberg likes these files to be large. Even with a wait time for accumulation, hitting the right size in time isn’t always possible. It’s common for Iceberg deployments to end up holding files that are smaller in size than optimal. This hurts query performance. Compaction is Iceberg’s mechanism for fixing this - it combines the small files together into larger files.

2. Snapshot expiration - Snapshot’s in Iceberg represent the state of a table at a given time and are required for Iceberg’s time travel feature (time travel allows you to view a table at its previous state in history). Whenever new data is written to an Iceberg table a snapshot is created so that time travel can revert back to the table state before the write. Over time, these snapshots build up and make querying increasingly slower and more compute-intensive. Snapshot Expiration mechanism was created to delete older ones and restore performance.

Issue 3. Lacking a single source of truth 🧭

Any copy of data introduces the possibility of inconsistency. Corruption, duplication, schema skew are but a few examples of what can happen when the same data point is written across many destinations. There are many ways to slice the Kafka -> Iceberg flow to reduce or promote duplication between the two systems. Each offers different trade offs and hence must be evaluated carefully.

Issue 4. Partitioning 🗂️

Both Kafka and Iceberg have the concept of a partition but they are vastly different in meaning.

• In Kafka: a partition is a separate log. It acts as unit of concurrency used only to parallelise processing.

• In Iceberg: a partition is an organisational structure imposed on data to allow efficient querying. Iceberg partitions group similar data together and label it so that queries that reference that data know which groups are relevant to the query and which are not.

Effective partitioning in Iceberg means the difference between reading every file stored to satisfy a query and reading only a few relevant ones. It can increase query performance by orders of magnitude so must be considered.

Solutions

Ok, enough problems. Onto the solutions. There are 2 general approaches to the Kafka -> Iceberg flow today:

• copy: those that simply copy data

• zero copy: those that integrate the two systems more tightly.

First I will look at the former: systems that treat Kafka as the source for an analytically focused Iceberg copy.

🧾 Copy based solutions

In general these have the following pros and cons:

✅ Pros:

• It is easy and generally not impactful to reorganise data in a copy - copies can have different partitioning, materialisation and other operations applied during the flow that make them better for analytical purposes without affecting the source Kafka.

• Simplicity - Copies in Iceberg are separate from the original data in Kafka making it easy to understand the flow and resulting data sets.

⚠️ Cons:

• Every copy of a dataset requires extra storage resources (and associated costs) and introduces the possibility for inconsistency.

• Copies involve the transfer of data from one system to another and so , due to the laws of physics, mean that the destination system (Iceberg) must lag behind the source system (Kafka). With Iceberg in particular, reducing this lag increases table maintenance so a careful balance must be maintained to achieve optimal resource usage and availability.

• Copy based solutions usually have a large maintenance overhead coming from external systems and practices that work alongside the main data transfer flow.

1. Kafka Connect

The Iceberg Kafka Connect Sink connector enables you to ingest records from Kafka topics and write them into Iceberg tables (on various storage backends) using the Iceberg table abstraction. As a core piece of the Apache Iceberg project it is open source, mature and feature rich supporting routing, schema evolution, table partitioning and external Iceberg catalogs (Hive, Glue, Nessie, etc.).

Its key advantage is integration with the Kafka Connect ecosystem. If you are already running Kafka Connect configuration is little more than a new library and some JSON. With that you get Connect’s scaling, fault tolerance, checkpointing, transforms (via SMT) and many other benefits. This is great if you have an existing Connect estate, however, if you do not have such an estate there is a steep learning curve in creating this platform before you can use the connector.

Unfortunately, like most connectors, the Iceberg connector considers its job merely to transport data. This means that table maintenance operations such as compaction and snapshot expiration must be handled separately. Also, due to the efficient data transfer focus there can be a significant lag between data creation in Kafka and availability in Iceberg (this is a common theme in copy solutions).

2. RedPanda Iceberg Topics

Iceberg Topics enables an additional flow in Redpanda to the regular log storage that persists topic data directly into Iceberg table format. Unlike the connect approach above this is handled within the broker without needing a separate connector/ETL process.

This closer integration means that RedPanda goes deeper into the downstream Iceberg tables providing features such as automated snapshot expiration and custom partitioning. RedPanda has also focused on the wider ecosystem, offering native integration with the largest Iceberg consumers (Snowflake/Databricks/AWS Glue) and their catalogs.

One current limitation of the RedPanda solution is the inability to backfill an Iceberg table from an existing Kafka topic. For instance, if your Kafka topic has a retention of 7 days and you enable Iceberg integration it will be a full week before the Iceberg table and Kafka topic are consistent. However, I spoke with the folks at RedPanda and they are actively developing a solution to this restriction and it’s due to be addressed in the near future.

RedPanda also provides other enterprise features needed to run an effective Kafka -> Iceberg solution such as a DLQ table for records that are incompatible with the destination Iceberg table and support for storage on all 3 major cloud providers.

The killer feature here is time to value, RedPanda have resolved many of the barriers to an effective Kafka -> Iceberg flow meaning you can be up and running in seconds, however, with enterprise features comes enterprise licensing and you will need one to use this feature .

3. Confluent TableFlow

Tableflow is a cloud based, managed Kafka -> Iceberg flow from Confluent Cloud similar to RedPanda’s above. If you are an existing Confluent Cloud user, you can materialize Kafka topics into open table formats such as Apache Iceberg or Delta Lake with a simple topic configuration.

TableFlow automates many of the common tasks needed when creating a Kafka -> Iceberg (or DeltaLake) pipeline including schema mapping / evolution, type conversions, compacting small files, publishing metadata to catalogs and many more. Like RedPanda, Confluent supports all major external catalogs (AWS Glue, Snowflake Polaris, Apache Polaris, Unity Catalog) but also provides the option to use its own catalog and storage for the true one-click up and running experience.

Confluent also provides other enterprise managed table features including compaction and snapshot expiration to keep tables clean and up to date. Furthermore, you can instruct TableFlow to do more than just make one to one copies of Kafka data, it supports Upsert/CDC modes to create materialisations based on keys/instructions.

The advantage of this solution is the maturity of the Confluent Cloud ecosystem. Confluent are able to leverage their mature cloud solution to create a “turn-key” experience that is trivial for existing Confluent Cloud customers. Unfortunately, this experience comes at a cost. TableFlow is significantly more expensive than its competitors today and locks you into the Confluent ecosystem.

4. WarpStream TableFlow

The newest entry into the Kafka -> Iceberg arena, Warpstream tableFlow is, like its namesake, a fully packaged solution for materialising Iceberg tables from Kafka data. Also like its Confluent counterpart, Iceberg tables created by Warpstream are fully managed right through data commitment, compaction, snapshot expiration and metadata management.

Where Warpstream’s solution differs is in it’s integration beyond the Confluent ecosystem. Warpstream TableFlow is a true BYOC system where you define source clusters (Kafka or Kafka-compatible from any provider) and designate where the resulting table data is stored (object storage buckets from any cloud provider either inside or outside Warpstream)

It supports integration with external catalogs and query engines. For example, you can register tables in catalogs (e.g. Glue, Snowflake, others) and provides guides on how to access this from the usual Iceberg engines (Trino, Spark, Clickhouse etc.).

Warpstream TableFlow also provides fine-grained data retention controls, allowing users to configure how long data is preserved at both the stream and table levels. Retention policies automatically manage snapshot expiration and file cleanup in object storage, ensuring that Iceberg tables remain compact, cost-efficient, and compliant with organizational data lifecycle requirements.

In summary, WarpStream’s TableFlow emphasizes simplicity, openness, and low-cost Iceberg materialization, while Confluent’s focuses on enterprise-grade governance, observability. There is significant overlap and it’s interesting to see such potentially competing products come under the same company umbrella.

5. AutoMQ Table Topics

Table Topics is a built-in AutoMQ feature that lets data sent to certain Kafka topics be automatically materialized into Apache Iceberg. Like RedPanda, you don’t need a separate connector or external streaming engine, and work with separate copies of the data for Iceberg and the Kafka log.

As above, AutoMQ handles the full Iceberg flow including file writing, metadata commits using internally managed services. All of these operations are coordinated inside the broker by services that piggy back on AutoMQ’s stateless, elastic and leaderless brokers. In AWS, AutoMQ is the only solution here that can integrate with the newer “S3 Tables” (a cloud provider managed data + catalog construct) so that the materialized tables use Amazon’s catalog and maintenance features (compaction, snapshot removal) and are queryable via Athena, etc. This is not yet available f

One early gotcha, however, is that it must be enabled at cluster deployment time and cannot be retroactively enabled like TableFlow and RedPanda. This means integration must be scheduled and managed in a much more intrusive way.

Like TableFlow AutoMQ’s implementation allows you to materialise tables using CDC/Upsert semantics and, in a similar way the data structures required for this are Schema Registry driven (usually Confluent Schema Registry or Aiven Karapace). Unlike TableFlow, AutoMQ is available outside of managed cloud services. The code behind its Table Topics (and all other features) is open source (Apache 2.0) and available to run on premise.

Ultimately, AutoMQ’s Table Topics is a powerful way to unify streaming and analytics, turning Kafka topics directly into queryable Iceberg tables without the need for separate pipelines or orchestration layers.

Summary of copy based solutions

We’re at the half way point 🏁 so let’s recap what we’ve seen so far:

Kafka Connect Iceberg Sink →open source and very flexible, if you can stomach its higher ops overhead.

Redpanda Iceberg Topics → Super easy setup for new topics/flows but cannot be imposed on existing data.

Confluent Tableflow → Enterprise-friendly but cloud only and vendor locked.

AutoMQ Table Topic → Best for tight cloud-native integration (eg. S3 tables) and open-source posture, but newer/less battle-tested.

⚡Zero Copy solutions

Up until now we have focused on solutions that maintain a clear separation between Kafka data and its Iceberg counterpart. These solutions generally force you to make an ugly choice between data freshness and Iceberg storage efficiency:

There is another class of solutions that address this trade off by integrating Kafka and Iceberg more closely. These typically share storage between Iceberg and Kafka in some way and eliminate copies. This provides reduced cost and improved consistency.

In general these have the following pros and cons:

✅ Pros:

• Reduced lag - Zero copy solutions streamline the path of data from ingest to Iceberg by sharing storage between the two.

• A Single Source of Truth - Any data written into a shared Kafka/Iceberg solution will be stored only once and accessed by both Kafka and Iceberg clients. This ensures total consistency between the two systems.

• Cost reduction - Only one copy of the data is stored removing not only duplicated storage costs but management and reformatting costs too.

⚠️ Cons:

• Restrictions - A tighter integration means that either side of the Kafka/Iceberg divide must take on aspects of the other. A good example of this is Iceberg partitioning. If a system is to read Iceberg data efficiently through the Kafka protocol it should be partitioned by Kafka partition and offset and not in a way more suited to analytical queries..

• Complexity - Integrated solutions require much closer management of the underlying data and associated read/write paths. This adds complexity and hidden behaviour. Usually this is internal complexity, however and these solutions often appear simpler to the wider world.

6. Bufstream

Bufstream is a Kafka-compatible streaming platform that can stream data directly from Kafka topics into Iceberg tables (i.e. broker-side materialization). Bufstream was developed with cloud native object storage as the primary storage and with a focus on schemas right from the beginning, excellent building blocks for an Iceberg solution. In Bufstream’s, parquet files are written to object storage by the broker on produce and shared by both Iceberg and Kafka readers, avoiding duplicating data and creating a true zero copy layout. The trade off here is that object storage is on the write path and adds producer latency. Bufstream typically experiences 3-5x higher end to end latency (P99) than traditional Kafka.

Bufstream handles the associated Iceberg metadata and maintenance operations are taken care of inline with Kafka (retention etc.) and Iceberg (compaction, snapshot expiration etc.) requirements. Bufstream also supports the various external Iceberg catalog types (REST catalogs, AWS Glue, Google BigQuery Metastore) however there are constraints here because the data is shared. The most important of these is that Iceberg data is read only and does not support operations that rewrite the underlying data files. Shared solutions must serve Iceberg data back to Kafka so rewrites of that data by external engines can have unexpected consequences.

As schemas are a first class citizen in Bufstream, the integration is schema-aware, interacting with a schema registry (especially for Protobuf schemas) to derive Iceberg schemas, enforce schema compatibility, and do semantic validation.

In the end, Bufstream design brings streaming and lakehouse paradigms together under a unified schema model, providing consistent semantics across ingestion, storage, and consumption, with a latency cost…

7. Aiven Iceberg Topics

Aiven’s solution adds native Iceberg support into Apache Kafka itself via a custom implementation of Kafka’s pre-existing tiered storage mechanism. This splits Kafka data into two sections, a “hotset” stored in Kafka format to handle the most recent data and serve most Kafka clients and a “coldset” of older data that is transferred to object storage and persisted as Iceberg. This architecture is zerocopy because these two sets are interoperable, with the “coldset” able to be recalled to Kafka duty as required. This simply takes Kafka’s existing tiered storage flows (Kafka <--> S3) and changes the format (Kafka <--> Iceberg).

Unfortunately the reliance on tiered storage introduces some lag into the Iceberg path. Tiered storage was designed to facilitate the cheap storage of rarely utilised data and so typically allows a configurable but usually significant amount (>24hrs worth) of data to accumulate in the “hotset” before it is tiered. The time taken for this accumulation is also the time behind real-time that the Iceberg data will be.

By design, Iceberg Topics establishes a boundary around what is Kafka’s responsibility and what is Iceberg’s. Unfortunately this means that compaction, snapshot expiration and other Iceberg maintenance operations are beyond the scope of the plugin and will need to be added externally.

Tiered storage is one of the few pluggable areas of the Apache Kafka project and Aiven have added Iceberg features by extending their industry standard open source tiered storage plugin to include Iceberg support. If you’re already using Aiven tiered storage, adopting Iceberg is just a simple update and config switch.

Aiven’s Iceberg Topics plugin elegantly leverages Kafka’s tiered storage to unify streaming and analytical use cases without duplicating data, maintaining zero-copy interoperability between Kafka and Iceberg.

8. StreamNative Ursa

Ursa is a “lakehouse-native” streaming engine from StreamNative, which provides Kafka-API compatibility but writes data to open table formats (Iceberg, Delta Lake) on object storage. It uses a leaderless, stateless architecture taken from Apache Pulsar: rather than Kafka’s traditional brokers with leaders and replicas. This is an architecture similar to the recent diskless topics addition to the Kafka world but it has been in Pulsar from the beginning.

As a recently developed, Kafka compatible, engine Ursa is missing some features on the Kafka side, most notably topic compaction and transactions. Both of these are commonly used so could affect Ursa’s adoption rate.

Ursa includes stream storage + columnar storage layers: it writes a write-ahead log (WAL) for fast ingestion and then commits data into Parquet files (archive files) for analytic reads. It integrates with the usual bunch of external catalogs and some more specialised ones (e.g. S3 tables).

The use of a WAL introduces a similar lag problem to Aiven’s solution and, with similar boundary definitions, Ursa will not automatically handle compaction and snapshot expiration internally.

Also in a similar vein to Aiven’s solution, Ursa supports the use of the Iceberg layer to serve Kafka tasks (a true zerocopy). It even has a clever indexing system that spans both WAL and Parquet data to increase the efficiency of this.

Ursa combines streaming and lakehouse concepts through a Pulsar-inspired, stateless design that writes data directly into open table formats without traditional brokers. Its dual-layer storage model enables zero-copy interoperability while maintaining strong performance for both streaming and batch workloads.

9. Streambased

Streambased takes a completely original approach to this problem. Like tiered storage, it maintains the concepts of “hotset” stored in Kafka and a “coldset” stored in Iceberg. Where it differs however is that Streambased surfaces the hotset via protocol translation without requiring any data movement. The flow is such that, at query time, the latest hotset data is fetched from Kafka, transformed to Iceberg and combined with the coldset (already Iceberg) before being surfaced to the client. In this way clients see the entire dataset available to them from the latest data produced a millisecond ago to the oldest data stored in Iceberg since the beginning of time.

The big advantage of this approach is that there is, by definition, zero lag. Because the hotset data is fetched on demand at query time and not surfaced via an asynchronous background process, Streambased can guarantee that every Iceberg query is working with the very latest data available to it.

Such an approach means differences in resource usage too. Streambased decouples the pure Kafka load (produce) from the load incurred converting from Kafka to Iceberg. Other solutions mentioned here run background processes to do this conversion meaning that, when Kafka load spikes, so does conversion load. With Streambased the conversion load is coupled to the Iceberg query load not the Kafka write load, isolating the 2 usages.

Like Aiven, Bufstream and Ursa. Streambased can also serve Kafka data from Iceberg. Streambased K.S.I. (Kafka Service for Iceberg), is a Kafka proxy that federates incoming fetch requests between the hotset in Kafka and the coldset in Iceberg. Data from either is surfaced as appropriate to Kafka clients.

In deployment, Streambased is also a significant departure from the other solutions in this article. Streambased does not seek to replace or modify an existing Kafka instance, instead it sits as a layer above requiring only the public consumer and admin protocols to function. This means Streambased can be deployed rapidly on top of managed Kafka like Confluent or Bufstream and on prem open source distributions.

Streambased’s architecture redefines zero-copy by translating Kafka data into Iceberg format on demand, merging data at query time for truly real-time analytics. This eliminates latency entirely.

Solution Summary

As can be seen from this overview, all solutions in this space have the required building blocks to create an effective Kafka to Iceberg solution. The decision as to which technology to adopt will likely be driven by external factors rather than any particular feature. For this reason I’ve outlined which solution you may choose given some of these factors below:

For open-source, DIY control → Kafka Connect Sink is flexible but ops-heavy. Aiven and AutoMQ provide open source versions for customization.

For vendor simplicity → Redpanda Iceberg Topics and Confluent TableFlow offer simple enterprise style integration if you can accept the vendor lock-in.

For maximum data freshness → Streambased’s query-time translation guarantees zero lag between Kafka and Iceberg, ideal for real-time analytics and observability workloads.

For maximum compatibility → Streambased requires only the Kafka wire protocol so can function with any number of existing Kafka deployments.

For schema-first organisations → Bufstream’s schema-aware design ensures consistency and easier evolution across streaming and lakehouse layers.

For cost-efficient scalability → Ursa, Bufstream and AutoMQ’s stateless brokers and shared object storage reduce infrastructure costs while maintaining Kafka-compatible performance at scale.

Conclusion

The convergence of Kafka and Apache Iceberg represents one of the most exciting frontiers in modern data infrastructure. While each technology was born to solve fundamentally different problems, the demand for real-time decision-making has forced these worlds together. The appetite for a unified data view is strong as evidenced by the widespread support that has come so quickly after Iceberg won the table format wars.

The industry’s range of approaches, from traditional copy-based pipelines to innovative zero-copy architectures, underscores that there is no single “right” solution, only a spectrum of tradeoffs between cost, complexity, freshness, and control. Copy-based systems like Kafka Connect and TableFlow deliver predictable, well-understood architectures but introduce lag and duplication, while zero-copy systems like Streambased, Bufstream, and Ursa push the limits of what’s possible with shared storage and dynamic federation.

In this evolving landscape, the unification of streaming and analytical data systems is no longer a distant vision, it’s quickly becoming the default expectation for modern data infrastructure.

Please find a summary table below of all solutions side by side:

Architectures grow organically, technologies and processes are chosen to suit a specific first case and, if successful, expand. The problem is that this is happening (and has been happening for the last 20yrs) concurrently across orgs across the entire business. The result: many disparate technologies connected by brittle pipelines.

For example, a simple retail company may have:

• Started with a MySQL database for its point-of-sale system

• Later added a SaaS CRM to track customer interactions

• Then adopted a cloud data warehouse to support online analytics.

• Then marketing spun up its own data pipeline into a separate BI tool

• SREs added metrics to one system and logs to another

• The CTO insisted on adopting the next big thing

• The data science team set up their own Jupyter cluster with custom ETL jobs pulling from production databases

• Shadow IT introduced spreadsheets and Google Sheets integrations as "glue" for missing workflows

• [these are all things that I have experienced, add your own in the comments ;-) ]

Meanwhile:

• Operations engineers maintain bash scripts that shuttle CSVs between systems

• The warehouse has nightly jobs failing silently

• Marketing dashboards rely on stale data

• Customer service can’t reconcile records between CRM and order history

• Data schemas become rigid and stale because of the impact of changes on downstream systems.

• …

This story has been played out time and time again and always has the same conclusion. Data infrastructure eventually buckles under the weight of scale, regulation, and business complexity.

Standardization is one of the key levers for regaining control when architectures have grown in an organic, fragmented way. Consolidating around standards breaks down silos, reduces complexity and increases scalability.

This is not a novel idea, most cutting edge operational teams have already standardized around Kafka and similar leading analytical teams have already standardized around Iceberg.

Clear but isolated standards only solve half the problem however, leaving the fundamental disconnect between operational and analytical systems intact.

The combination of Kafka and Iceberg, on the other hand, is incredibly powerful because it finally resolves the decades-long tension between real-time operations and durable analytics. Used separately they still leave us with duplicated datasets, costly ETL jobs, schema mismatches, and latency compromises. Together, they create a unified architecture where streaming and batch pull together rather than against each other. An architecture much more powerful than the sum of its parts.

Let’s explore an architecture based exclusively on Kafka and Iceberg with the aim of achieving the gold standard in both operational and analytical arenas. Including:

• Cheap long term retention - Store data for as long as it’s required without having to worry about mounting storage costs and maintenance overheads

• A single source of truth - Achieve total consistency across the entire data estate

• <100ms access - Clients on both sides of the operational/analytical divide should work with the latest data available

• Easy evolution - Data should be able to evolve with the business, adopting volume change, schema evolutions and changing business needs at will.

The new (old) architecture

There is an architecture that fulfils these requirements and it’s actually a well understood, production proven one: The lambda architecture.

For many (including me) the Lambda architecture (proposed by Nathan Marz in 2011) was a first introduction to real-time data, stream processing and concepts beyond the high volume, batch based infrastructures that we all grew up on.

As a quick recap it consists of a batch layer that provides a long term (months/years), high lag (hours/days) view that is topped up with a speed layer that provides short term (<7 days) low lag (sub-second) data. A complete picture of the data can be built up by combining data from one or more of these layers.

The issue with this architecture is that Speed and Batch layers typically have vastly different access patterns and concepts. This makes Mixed Applications (where the most power lies) complex to develop, manage and evolve. For example, Speed Layer applications must routinely address situations like late arriving, out of order data etc., concepts that simply aren’t present in the Batch layer.

The fundamental problem here is that batch and speed speak very different languages, but what if they didn’t? What if they both spoke Iceberg?

Suddenly both our batch and speed layers speak Iceberg and unifying both data flows becomes a trivial problem

In our new architecture both speed layer and batch layers are represented as Iceberg tables and the combination of them both can be achieved with a simple union:

SELECT *
FROM
  speed_layer
UNION ALL
  batch_layer

Traditionally this union was done by a complex client system (Apache Beam anyone?) but recently speed critical OLAP databases like Apache Pinot are filling this role. However having everything in Iceberg means no client system commitment must be made and any processing engine supporting Iceberg (Spark/Dremio/Trino etc etc..) can be used.

So how do we implement this?

#NoETL

The traditional way to solve this problem was to have an ETL process (Kafka Connect is a great one) that sits between Kafka and Iceberg to copy the data across into a second, analytical dataset. Unfortunately this approach involves compromising on the Single Source of Truth and <100ms goals:

• Copies of the data - Any ETL process copies data from the operational layer (Kafka) to the analytical layer (Iceberg). Since these layers evolve independently, inconsistencies can arise. For example, if Kafka records are accidentally duplicated in Iceberg, can Iceberg still be trusted? Or if Iceberg enforces business rules (e.g., amount ≥ 0) that Kafka doesn’t, can Kafka still be trusted?

• High latency - Copy-based approaches also add delay. To store data efficiently in Iceberg, records must be batched into larger chunks. This batching often takes minutes to hours (typically ~15 minutes), during which the data is unavailable in Iceberg. For more detail see my LinkedIn post on this here

Not only that, but it forces you to contend with some troubling side effects:

1. High maintenance - ETL generally writes small and inefficient packets of data to Iceberg continuously and incur the maintenance overhead that comes along with this is large. (I’ve written previously on this here). Not only that but the operational and analytical realms often have different concepts making governance processes like schema evolution challenging.

2. High storage costs - ETL requires keeping two full copies of the data: one in Kafka for operations, and another in Iceberg for analytics. This duplication inflates storage bills dramatically when volumes are high.

It has to be a view

Surfacing the Speed Layer as Iceberg is not as simple as it seems. This process must retain the ultra low latency access expected from the speed layer but Iceberg is not optimised for this at all.

ETL processes can not achieve this, instead we can rely on a trusted mainstay of database systems: Logical views.

Logical views involve result sets computed at query time rather than pre-computed and so are guaranteed to work on the latest available data and achieve the latency goals we are looking for. What’s more, they needn’t be any more complex than their ETL counterparts, it’s the same processing taking place just executed at a different point.

This creates an Iceberg-like layer over Kafka that can be accessed by processors in the same way as any other Iceberg store. When an Iceberg query is executed, the view calculates the data from Kafka required to satisfy the query, fetches it, transforms it to an Iceberg format and returns it. This all happens on demand, with the latest available data from Kafka. For more information see my recent LinkedIn post on this here

This comes with a number of advantages over ETL:

• Always up to date - In a logical view data is fetched at query time, giving access to the complete latest-millsecond available data. Connectors/materialized views involve copies of the data that may lag behind the source.

• Evolves with the source data - a logical view is guaranteed to be consistent with the source data meaning that any changes in the source data (schema evolutions, repartitioning etc.) are immediately reflected in the view. Other approaches require an expensive and time consuming replay of older data before evolutions are reflected. I’ve written on this previously here

• Zero Copy - only a single copy of the data need be stored in Kafka, no separate Iceberg copies.

Putting it all together

The final step towards our complete architecture is to create a path to transition data from speed layer to batch layer. As both layers are represented by Iceberg tables in this architecture this can be as simple as an insert statement:

INSERT INTO
  batch_layer
SELECT *
FROM
  speed_layer

Care must be taken to balance the sizes of Batch and Speed layers to maintain efficiency. However, as the transition process is just another Iceberg operation, it can be invoked at any point and there is no down time incurred. The transition process is as follows:

1. Initially the majority of data exists in the batch layer with only a small “head” held in the speed layer:

   

2. New data is written to the speed layer only, increasing the share of data it is responsible for serving:

   

3. When a threshold is reached (usually driven by Kafka topic retention) a move process is triggered to move data from speed layer to batch:

   

   Note that this transition process involves moving a large amount of speed_layer data efficiently in large optimized chunks, thus avoiding the small files (compaction) and snapshot expiration issues usually associated with streaming to Iceberg

4. With this process complete we return to the initial state with the batch layer serving the majority of data and repeat the process again and again:

   

Conclusion

By unifying Kafka and Iceberg through a logical, view based, architecture we can finally dissolve the long-standing divide between operational and analytical systems. Instead of maintaining parallel infrastructures, duplicating data, and fighting constant trade-offs in latency and consistency, organizations can achieve a single, coherent data layer that is both real-time and durable.

This model not only simplifies architectures and reduces cost, but also empowers teams to build richer, mixed applications on top of a unified source of truth. In many ways, it delivers on the original promise of the Lambda architecture: speed and batch working together but with the elegance and efficiency of a single language: Iceberg.

We’ve spent the past 2yrs building this architecture at Streambased. Want to try it out? Drop me a DM:

Answers have ranged from a super simple Kafka connector to multi-stage, Flink-based pipelines but that’s not what this post is about.

Instead, we’re going to address the elephant in the room and ask the real burning question: “What happens when your Kafka data gets there?”.

If you’re a mature Iceberg user, chances are you are butting up against some fundamental Iceberg challenges that are only exacerbated by streaming data sources.

In this post we’re going to look at 3 of the most common and explore a novel approach to address them.

1. Snapshot Expiration

Time travel is a flagship feature of Apache Iceberg that allows queries to be executed against a table state at any point in the table’s history. It’s powered snapshots, a “snapshot”: a unique piece of metadata that points to all the data in the table at that time.

Imagine a situation where I insert 2 rows and then 30 seconds later insert a further 2. This action created 2 snapshots, the first pointing to a table with 2 rows and the second pointing to a table with 4.

Each snapshot is given a unique id, allowing you to time travel back to the table state at the snapshot:

SELECT * FROM someTable FOR VERSION AS OF 2188465307835585443 -> 2 rows
SELECT * FROM someTable FOR VERSION AS OF 4583289324735846932 -> 4 rows

Imagine, for instance, that you wrote every message from an event stream individually to an Iceberg table via an “INSERT INTO” statement (a really bad idea). By default the number of snapshots would be equal to the number of messages (potentially millions per second)!

Thankfully snapshots are metadata only, so the cost of creating one is not huge, however frequent writes etc. create a lot of snapshot and hence a lot of metadata that must be managed. If snapshots are allowed to grow uncontrolled you can suffer from some serious side effects:

• Delayed query planning - Processing metadata is the first stage in any Iceberg query, the more metadata, the longer this process takes.

• Slower rollbacks - Iceberg rollbacks must traverse the snapshot list to find the correct snapshot to roll back to. The larger this list the slower this operation. Performance can be further hit if orphaned snapshot metadata is pruned during the rollback.

• Degraded maintenance operations (e.g. compaction)

• Increased resource consumption

The solution to this problem is Snapshot Expiration, a manually triggered Iceberg process where older snapshots are removed.

Remember however, that Iceberg is only a format so cannot do the expiration work by itself.

Instead it must rely on an external engine like Spark or Trino (or any number of other vendors) to perform it. Furthermore this is not an automated background process you can switch on and forget about, it’s a repeated batch job that must be scheduled and managed separately from read/write jobs.

Expiring a Snapshot can create its own problems! Once expired, Iceberg loses the ability to time travel to that snapshot so any queries run at that snapshot will fail.

In other words, your Iceberg administrator has to carefully balance metadata size against time travel capability manually in a never ending compromise.

2. Compaction

Snapshotting is not the only feature that suffers damage in the presence of small inserts in Iceberg. Such write patterns can also create many small files that wreck your general Iceberg query performance.

Standard Iceberg behavior dictates that every completed insert statement creates at least one new file.

Returning to our previous scenario where 2 rows are inserted and then 30 seconds later a further 2, would create 2 new Parquet files to store their respective data in.

Technically we still live in the world of Big Data and a file containing 2 rows is very small. The last thing you want is a table made up of millions of tiny files that:

• Reduce read performance - To scan the data in the table every file must be opened, read and closed. This can increase query times by a missive amount.

• Increase storage requirements - Files typically contain extra metadata, repeating this many times across many small files results in bloated inefficient use of disk space.

• Worsen compression

• Increase costs via more separate access requests - If your filesystem incurs extra costs for open/close operations (such as Amazon S3) then more files means more of these costs.

• Complicate parallelization - Small files mean processing engines must make intelligent decisions around the way in which they combine files for parallelism. It is harder to define the optimum use of resources with this restriction.

The solution is to accumulate data before writing to Iceberg in bugger chunks but this can incur significant delay. To put this in perspective, the Parquet project (Iceberg’s most popular file format), recommends file sizes between 128Mb and 1Gb. Kafka messages are typically around 1Kb - 10Kb. Imagine a Kafka topic with 20 partitions receiving 5Mb/s. This would map to 250Kb/s per file (1 file per partition) and take 66 minutes to fill a 1Gb Parquet file. Not very real time, is it?

To counter this, streaming pipelines typically write inefficient small files into Iceberg first and combine them into larger files later. A process called Compaction. This is an essential housekeeping process and is manually triggered and monitored by your Iceberg administrator who must determine and orchestrate the data accumulation required to achieve freshness/efficiency balance required.

But again! Remember that Iceberg is only a format so cannot do the work by itself. 💡

Like snapshot expiration, It must rely on an external engine to perform compaction and this is not an automated background process.

Compaction determines the balance between freshness, performance, and cost and must be constantly maintained. Administrators should be constantly determining what the optimal file size should be and scheduling compaction when resources, a non trivial overhead.

3. Partition evolution

Partitioning is the fastest way to increase your Iceberg query performance by 100x or more.

It involves the organisation of data into subsets based on one or more column values. Query engines can take advantage of this organisation to only read the subsets relevant to the queries being executed.

For instance, given a table that is partitioned by timestamp and the query:

SELECT * FROM table WHERE timestamp = '2025-07-01';

The query engine can “prune” away any other timestamps, greatly reducing the amount of data read to complete the query (and making it go a lot faster!).

As you can tell above, partitioning needs to be matched to the query to work effectively. If this is not the case it can actually reduce performance and cause serious problems including:

• Reduced parallelism - Processing engines can parallelize by partition, reducing partitioning can reduce the maximum parallelization possible.

• Under-utilization of system resources

• Poor join performance - Joins can often be optimized by working with partitions rather than the full table. With poor partitioning this is not possible.

But what happens when you query patterns change? To maintain performance, partitioning must change along with them.

Iceberg supports “Partition Evolution” for this. Using a simple set of commands you can change the partitioned columns of your table to better match your query profiles.

Like most Iceberg maintenance operations this is a metadata only operation and so does not require the underlying data to be rewritten… yet.

Partition evolution applies only on data written after the partition spec has changed. Any earlier data still has the earlier spec applied and, whilst still available to queries, will not benefit from the increased performance promised by evolution.

The solution to this problem is to rewrite old data to match the latest spec.

Once more however, remember that Iceberg is only a format so cannot do the work by itself. 💡

It must rely on an external engine to perform the required data rewrite. But this is a resource intensive process that can take a long time (think hours) to perform.

Iceberg administrators must pay attention to partitioning. At first this appears simple but, as volumes increase and query patterns change it can become complex, resource intensive and unpredictable.

Streambased to the Rescue

Up until now we’ve laid out a small selection of the administrative issues that plague Iceberg tables. It’s these issues that prompted us at Streambased to create I.S.K. (Iceberg Service for Kafka), an altogether simpler approach to streaming data via Iceberg.

Streambased performs an end run around all of this by surfacing Iceberg data as a logical projection rather than doing any physical movement of data.

In the Streambased approach the data that backs your Iceberg table remains in Kafka and ephemeral metadata is created on top of it. To clients, it appears as a normal Iceberg table but, when queries are executed, the data is read from Kafka, mapped and presented in a format clients are expecting

No data movement ahead of query time means no small files, no snapshots and no uncomfortable partitioning decisions. Instead, slice your data the way you want to at the point at which it is read. The need for Compaction is completely negated as the source system (Kafka) is responsible for the data layout, not Iceberg.

To maintain performance, Streambased indexes the data in Kafka as it is written or read for operational purposes. These indexes allow Streambased to mimic Iceberg features such as:

• Partitioning: Using indexes, Streambased can target sections of Kafka data that are relevant to particular queries creating a similar outcome to Iceberg’s tradition partitioning. The advantage of taking this approach is that indexing is not tied to the physical layout of files on disk and so require no rewrite and no delays.

• Snapshots: Snapshots in Streambased are also related to indexing. Kafka data is already indexed by timestamp and other insert markers so identifying the sections relevant to a particular time period is an easy task. This means Streambased can impose any number of snapshots on the data as required by the clients and that these imposed snapshots can not and need not ever be expired..

Streambased repositions the source system (Kafka) as the source of truth for both operational and analytical applications, removing the need for intermediate data stores and the expensive maintenance operations that come along with them.

Conclusion

Ultimately, as organizations continue to embrace streaming architectures and real-time analytics in their lakehouse, the operational complexities of managing Iceberg in its traditional form become increasingly apparent.

Streambased offers a refreshing rethink, one that aligns better with the dynamism of modern data flows by abstracting away the burdens of compaction, snapshot expiration, and partition evolution.

With Streambased, not only is your Kafka data instantly accessible by any engine that supports iceberg, but you also don’t have to deal with any of the complexities of managing it.

Subscribe now