Stream Processing refers to the real-time processing of data-in-motion, meaning near-real time / real time computations of unbounded streams of records (usually small sizes), dynamic data continuously generated by multiple data sources. Continuous streams of data sources could be sensor events, website users activity, credit card transactions,.. etc.
Stream Processing frameworks are distributed architectures able to handle/process instantly continuous streams of data, handling the state of the operations and also being able to recover in case one of the machines in the cluster fails. This course goes in detail into 2 existing streams processing solutions : Apache Kafka KSQL & Kafka Streams API and Apache Spark Structured Streaming but as well into the basics of streams – using Apache Kafka (understand partitioning, how data is consumed from a stream,…).
In this course you will understand:
– Streams concepts
– Stream processing vs batch processing
– Relevant use cases and the architectures used
– Which are the most important components/features of a stream processing
solution/architectures
– Which are the most known stream processing solutions and how they
differentiate
– Kafka Streaming (KSQL and Kafka Streams API)
– Spark Structured Streaming
– How to create a solution based on KSQL and Spark Structured Streaming
This course is designed for solution architects, data engineers, machine learning engineers, software engineers and data scientists with a basic knowledge of scalable data processing techniques such as Hadoop, MapReduce, etc.
This course is introductory level.
Pre-requisites: a good understanding of IT distributed systems, understanding of Hadoop concepts: mainly HDFS, Apache Spark knowledge a plus, Scala basic level
Course Features
- Lectures 24
- Quizzes 0
- Duration 3 Zile
- Skill level All levels
- Language Romana
- Students 12
- Assessments Yes
-
Ziua 1
- Fundamentals of streams with Apache Kafka & KSQL
- Learn the fundamentals of streams of data, including how to work with the Apache Kafka ecosystem, data schemas, ApacheAvro, Kafka Connect and REST proxy, KSQL
- Understanding the way streams are stored in distributed systems. Storing the stream in a distributed manner will help the stream computation to be scaled out to a distributed cluster for filtering and transforming the streams
- Partitioning strategies for streaming topologies
- Understand concept of order and time in streams
- How data can be consumed from a stream – what means parallel processing of streams data
- Streams processing using KSQL
- Working with streams & tables
- Understanding time concept in stream processing
- Late events processing
- Windows/aggregates
- State handling – how the KSQL architecture handles state and failures
-
Ziua 2
- Streaming architectures/solutions overview (with use cases) – Kafka Streams, Flink, Spark Streaming, Storm, Samza
- Spark Structured Streams Processing (using Kafka)
- Intro Apache Spark: understand architecture and concepts: RDDs vs Dataframes/Datasets
- Concepts of Structured Streaming
- Stream DF’s
- Queries on streams
- Triggers
- Output modes: Console, File, Memory, Kafka
- Checkpointing – understand how Spark saves the state of the persistent queries and how handles failure
- Window aggregations
- Handling of late data – watermarking
-
Ziua 3
