Sure! Obviously the logical view can only surface what's available in Kafka so you would be looking at a maximum of 2hrs available via I.S.K. You're a couple of posts ahead but one thing we've been working on at Streambased is an all Iceberg Lambda style approach. The batch layer is traditional Iceberg on S3 etc. and the speed layer is I.S.K. This circumvents the main issue with Lambda in that you have to design applications to address both speed and batch (in this case they're both Iceberg) and also addresses Iceberg maintenance pains as you can schedule a 2hrly job to copy from speed to batch (and avoid small files, snapshots etc.).
Look out for some future posts (and demos) coming on this.
Almost! You still have to bring the processing engine (Trino/Spark etc.) but you can work directly with Kafka instead of having to stage it somewhere first.
Can this work if my Kafka server data is expired every 2 hours?
Sure! Obviously the logical view can only surface what's available in Kafka so you would be looking at a maximum of 2hrs available via I.S.K. You're a couple of posts ahead but one thing we've been working on at Streambased is an all Iceberg Lambda style approach. The batch layer is traditional Iceberg on S3 etc. and the speed layer is I.S.K. This circumvents the main issue with Lambda in that you have to design applications to address both speed and batch (in this case they're both Iceberg) and also addresses Iceberg maintenance pains as you can schedule a 2hrly job to copy from speed to batch (and avoid small files, snapshots etc.).
Look out for some future posts (and demos) coming on this.
Great post! Sounds like ISK is almost a DB engine with iceberg as the file format.
Almost! You still have to bring the processing engine (Trino/Spark etc.) but you can work directly with Kafka instead of having to stage it somewhere first.