"Streaming Data Mesh" OReilly. Currently writing his second book "Streaming Databases - Supporting Monolithic Data Engineers
hubertdulay.substack.com (https://hubertdulay.substack.com?utm_medium=podcast)
In this podcast Ralph and I interview a former colleague of mine, Kai (https://www.linkedin.com/in/kaiwaehner/), who has extensive experience in the data streaming and real-time events space. Kai highlights the top five trends for data streaming with Kafka and Flink, including data sharing, data contracts for governance, serverless stream processing, multi-cloud adoption, and the use of generative AI in real-time contexts. We discuss the role of generative AI in providing accurate answers and the importance of real-time data integration for contextual recommendations, using the example of travel and flight cancellations. We also delve into the role of Flink as a stream processor in ensuring the accuracy and freshness of data for semantic searches and generative AI applications.
We also delve into the idea of streaming databases and whether the market is ready to embrace them. We discuss the need for data contracts and data governance to understand the flow of data through systems, as well as the responsibility of the data engineering team in creating embeddings. We also discuss integrating large language models with other applications using technologies like Kafka and provide examples of how generative AI can be integrated into existing business processes. The interview touches on the concept of a "lake house" and the separation of compute and storage for real-time analytics. The guest also highlights Confluent's approach to building Kafka in a cloud-native way and their focus on the streaming side, while emphasizing the need for accessible stream processing solutions for ordinary database users.
Get full access to SUP! Hubert’s Substack at hubertdulay.substack.com/subscribe (https://hubertdulay.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)
28 Nov 2023
Interview with Materialize - Consistency
In this podcast, we interview Arjun Narayan (https://www.linkedin.com/in/arjunravinarayan/), Frank McSherry (https://github.com/frankmcsherry/), and Nikhil Benesch (https://www.linkedin.com/in/nikhilbenesch/) from Materialize. Ralph and I are writing a book on streaming databases and seeking expert insights from Materialize on topics rarely discussed in the field. We begin by exploring the distinction between operational and analytical workloads, highlighting the importance of real-time or near-real-time results for operational tasks. We further delve into the significance of consistency in operational workloads and the challenges of using eventually consistent systems. The guests caution against relying on eventually consistent stores and databases, stressing the value of consistency in certain domains like payments.
We focus on the concept of time in differential data flow, explaining how revisions provide a better understanding of time in this context. Consistency is highlighted as crucial in temporal joins, especially for mathematical operations and data enrichment. Overall, we emphasize the importance of real-time workloads, consistency, and integration in operational systems.
SUP! Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Get full access to SUP! Hubert’s Substack at hubertdulay.substack.com/subscribe (https://hubertdulay.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)
22 Nov 2023
Filipinos in Tech - Marlow and Ron
In continuing the Filipinos in Tech series, in this episode, I interview Marlo Carrillo and Ron Guerrero currently at Databricks but previously from Cloudera. We reflect on the significance of the Balikbayan box, symbolizing resilience and the importance of remembering their roots. We share personal and emotional stories of their own families' journeys to America, the struggles they faced, and the sacrifices made for a better life. We also discuss the challenges of growing up Filipino in different communities, feeling different, and trying to find connections. We highlight how Filipinos assimilate into new cultures while holding onto their heritage, and how language can be a marker of identity and assimilation. The episode explores the immigrant experience and the complexities of belonging to multiple worlds.
In addition to discussing our immigrant experiences, we focus on the impact of technology on the Filipino community. We speculate that more Filipinos will join the technology field in the future, including their family members. We discuss the preference for social and personal interactions that Filipinos may have, which could potentially explain the underrepresentation of Filipinos in the tech industry. We express gratitude towards America and its opportunities while acknowledging the unique charm of the Philippines. We also talk about retirement plans and the possibility of returning to the Philippines, with some expressing a desire to visit rather than permanently relocate.
SUP! Hubert’s Substack is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Get full access to SUP! Hubert’s Substack at hubertdulay.substack.com/subscribe (https://hubertdulay.substack.com/subscribe?utm_medium=podcast&utm_campaign=CTA_4)
Reach and audience
Public platform figures. Ratings count people who left a rating, not total listeners.
Apple Podcasts (US)
5.0 / 5
1 ratings
Host of Hubert's Podcast?
Claim your podcast to manage its listing and keep your show details accurate.
Pod Engine is an independent podcast discovery and analytics service and is not affiliated with or endorsed by this podcast. Artwork and show content belong to their owners. Full legal notice.
Explore this show Podcast research with Pod Engine
