Apache Flink's
dataflow programming model provides event-at-a-time processing on both finite and infinite datasets. At a basic level, Flink programs consist of streams and transformations. “Conceptually, a stream is a (potentially never-ending) flow of data records, and a transformation is an operation that takes one or more streams as input, and produces one or more output streams as a result.” Apache Flink includes two core APIs: a DataStream API for bounded or unbounded streams of data and a DataSet API for bounded data sets. Flink also offers a Table API, which is a SQL-like expression language for relational stream and batch processing that can be easily embedded in Flink's DataStream and DataSet APIs. The highest-level language supported by Flink is SQL, which is semantically similar to the Table API and represents programs as SQL query expressions.
Programming Model and Distributed Runtime Upon execution, Flink programs are mapped to streaming
dataflows.
HDFS,
Apache Cassandra, and more.
State: Checkpoints, Savepoints, and Fault-tolerance Apache Flink includes a lightweight fault tolerance mechanism based on distributed checkpoints. A user can generate a savepoint, stop a running Flink program, then resume the program from the same application state and position in the stream. Savepoints enable updates to a Flink program or a Flink cluster without losing the application's state . As of Flink 1.2, savepoints also allow to restart an application with a different parallelism—allowing users to adapt to changing workloads.
DataStream API Flink's DataStream API enables transformations (e.g. filters, aggregations, window functions) on bounded or unbounded streams of data. The DataStream API includes more than 20 different types of transformations and is available in Java and Scala. A simple example of a stateful stream processing program is an application that emits a word count from a continuous input stream and groups the data in 5-second windows: import org.apache.flink.streaming.api.scala._ import org.apache.flink.streaming.api.windowing.time.Time case class WordCount(word: String, count: Int) object WindowWordCount { def main(args: Array[String]) { val env = StreamExecutionEnvironment.getExecutionEnvironment val text = env.socketTextStream("localhost", 9999) val counts = text.flatMap { _.toLowerCase.split("\\W+") filter { _.nonEmpty } } .map { WordCount(_, 1) } .keyBy("word") .timeWindow(Time.seconds(5)) .sum("count") counts.print env.execute("Window Stream WordCount") } }
Apache Beam - Flink Runner Apache Beam “provides an advanced unified programming model, allowing (a developer) to implement batch and streaming data processing jobs that can run on any execution engine.” The Apache Flink-on-Beam runner is the most feature-rich according to a capability matrix maintained by the Beam community. data Artisans, in conjunction with the Apache Flink community, worked closely with the Beam community to develop a Flink runner.
DataSet API Flink's DataSet API enables transformations (e.g., filters, mapping, joining, grouping) on bounded datasets. The DataSet API includes more than 20 different types of transformations. The API is available in Java, Scala and an experimental Python API. Flink's DataSet API is conceptually similar to the DataStream API. This API is deprecated at Flink version 2.0
Table API and SQL Flink's Table API is a SQL-like expression language for relational stream and batch processing that can be embedded in Flink's Java and Scala DataSet and DataStream APIs. The Table API and SQL interface operate on a relational Table abstraction. Tables can be created from external data sources or from existing DataStreams and DataSets. The Table API supports relational operators such as selection, aggregation, and joins on Tables. Tables can also be queried with regular SQL. The Table API and SQL offer equivalent functionality and can be mixed in the same program. When a Table is converted back into a DataSet or DataStream, the logical plan, which was defined by relational operators and SQL queries, is optimized using
Apache Calcite and is transformed into a DataSet or DataStream program. ==Flink Forward==