Stream Processing Your Way with Apache Flink?

Apache Flink has become the standard piece of stream processing infrastructure for applications with difficulttosatisfy demands for scalability, high performance, and fault tolerance, all while managing large amounts of application state.

The key to demystifying Apache Flink is to understand how the combination of stream processing plus application state has influenced its design and APIs. A framework that cares only about batch processing, or one that performed only stateless stream processing, would be much simpler.

We'll explore how Flink's managed state is organized, and how this relates to the programming model exposed by its APIs. We'll look at checkpointing: how it works, the correctness guarantees that Flink offers, how state snapshots are organized on disk, and what happens during recovery and rescaling.

We'll also look at watermarking, which is a major source of complexity and confusion for new Flink developers. Watermarking epitomizes the requirement Flink has to manage application state in a way that doesn't explode as those applications run continuously on unbounded streams.

You'll leave with a good mental model of Apache Flink, ready to use it in your own stateful stream processing applications.

About Tim Berglund

Tim serves as the VP of Developer Relations at IBM, where he and his team work to make certain parts of the IBM portfolio more accessible to developers. He is a regular speaker at conferences and a presence on YouTube explaining complex technology topics in an accessible way. He lives with his wife in Arvada, CO. He has three grown children, three step-children, and four grandchildren, with a fifth on the way.

More About Tim »