Real-Time Data to AI Workflows: Kafka, Elasticsearch & Observability in Action

The Elastic Seattle User Group & Seattle Apache Kafka Meetup are partnering for a joint meetup on Monday, March 16th. We'll have presentations from followed by food, refreshments, and networking. 📅 Date & Time:Monday, March 16th from 4:30-6:30pm PST 📍Location:AWS Skills Center (across the street from its former location at the Amazon Kiro building) Amazon Oscar building1007 Stewart St2nd floorSeattle, WA 98101 🚗 Parking: SpotHero - Book a parking spot in advance 🪧 Arrival Instructions: Guests will go upstairs to the AWS Skills Center on 2nd floor. It’s directly up the stairs or elevator or they can use escalator and just follow floor signage to navigate to the entrance.Check-in with our guest reception team and then be directed to the rooms. Even though the Skills Center closes at 5pm, they will be open for the user group. 📝 Agenda: 4:30pm Doors open5:00 pm: One Does Not Simply Query a Stream - Viktor Gamov, Principal Developer Advocate, Confluent5:45 pm: : AI-powered workflows using Elastic Observability (O11Y) - Peter Simkins Sr. Solutions Architect at Elastic6:30 pm: Event ends 💭Talk Abstracts:One Does Not Simply Query a Stream - Viktor Gamov, Principal Developer Advocate, Confluent Streaming data with Apache Kafka® has become the backbone of modern applications. While streams are ideal for continuous data flow, they lack built-in querying capabilities. Unlike databases with indexed lookups, Kafka’s append-only logs are designed for high-throughput processing—not for on-demand queries. This necessitates additional infrastructure to query streaming data effectively.Traditional approaches replicate stream data into external stores: relational databases like PostgreSQL for operational queries, object storage like S3 accessed via Flink, Spark, or Trino for analytics, and Elasticsearch for full-text search and log analytics. Each serves a purpose—but they also introduce silos, schema mismatches, freshness issues, and complex ETL pipelines that increase system fragility.In this session, we’ll explore solutions that aim to unify operational, analytical, and search workloads across real-time data. We'll demonstrate stream processing with Kafka Streams, Apache Flink®, and SQL engines; real-time analytics with Apache Pinot® ; search capabilities with Elasticsearch; and modern lakehouse approaches using Apache Iceberg® with Tableflow to represent Kafka topics as queryable tables. While there's no one-size-fits-all solution, understanding the tools and trade-offs will help you design more robust and flexible architectures. AI-powered workflows using Elastic Serverless Observability (O11Y) - Peter Simkins, Sr. Solutions Architect, Elastic Elastic’s Serverless O11Y platform unlocks real-time visibility and action from streaming data—no infrastructure headaches. Ingest logs, metrics, and traces directly from Kafka and other sources, then automate investigations, trigger actions, and orchestrate responses across your data landscape.Streams is Elastic’s new agentic AI capability: it auto-partitions, parses, and surfaces significant events from noisy logs, transforming them into clear signals for rapid troubleshooting. Instantly correlate operational and application data, accelerate root cause analysis, and power AI-driven workflows—all within a unified, scalable platform.Elastic Workflows enable rules-based automation: set up triggers for anomaly detection, auto-remediation, or custom business logic. Connect to external systems, run playbooks, and drive end-to-end observability—all from a single platform.Serverless O11Y integrates seamlessly with Kafka, Flink, and Iceberg, enabling end-to-end observability and search across your real-time data pipelines.Live demo: See Elastic Streams and Workflows in action—AI-powered log analytics, event detection, automated incident response, and real-time alerting for modern data pipelines.