Open table formats like Apache Iceberg have become the backbone of the lakehouse, but streaming data still arrives with fragmented metadata and governance: topics live in messaging systems, schemas live in registries, and tables live in catalogs—often stitched together by connectors and convention.

In this talk, we propose a Streaming Data Catalog pattern for the Iceberg ecosystem: a metadata layer that treats a continuous stream as a first-class catalog object—on par with tables—while maintaining an explicit stream↔table linkage (typically 1:1) so operational producers and analytical engines share a single, governed representation of the same data.

We’ll cover the core metadata model (catalog / namespace / stream), the streaming-specific metadata that tables alone don’t capture (retention, offset-to-file mapping, consumer state, schema evolution contracts), and how the catalog becomes the “brain” that keeps protocol gateways stateless and interoperable.

We’ll also discuss practical interoperability: federating with existing schema registries, synchronizing with external metastores/data catalogs, and enabling multi-protocol ingestion (including lightweight HTTP/gRPC ingestion when deploying a full broker stack isn’t desirable). The goal is to spark a community conversation on standardizing streaming semantics around Iceberg tables—without reinventing the lakehouse.

David is a Developer Advocate at StreamNative and distinguished Apache Pulsar committer, recognized globally for his expertise in real-time data streaming, messaging systems, and big data technologies. As author of "Pulsar in Action" and co-author of "Practical Hive," he has established himself as a leading voice in the streaming data ecosystem.
An accomplished international speaker, David presents at conferences worldwide on big data, streaming technologies, and agentic AI, sharing insights that enlighten audiences globally. His technical contributions extend beyond speaking, as he actively contributes to Apache NiFi and maintains his committer status on Apache Pulsar, directly advancing these critical open-source platforms.
David's career spans key leadership roles at industry-leading companies. He previously served as Principal Software Engineer on Splunk's messaging team, specializing in real-time data analytics. Before that, he held Director of Solutions positions at influential big data startups Streamlio and Hortonworks, implementing enterprise-scale solutions using Hadoop, Hive, and Kafka for Fortune 500 companies globally.
Through his technical expertise, thought leadership, and community engagement, David continues shaping the future of real-time data streaming technologies.