As we understood all the moving parts such as logstash, kafka, spark streaming, hbase etc in building streaming data pipelines, let us see how we can integrate all these things together and come up with solution. Steps: Run Genlogs Script to generate retail e-commerce platform logs Use Logstash to transform logs into JSON format and …
“If data is the lifeblood of high technology, Apache Kafka is the circulatory system in use at LinkedIn. We use Kafka for moving every type of data around between systems, and it touches virtually every server, every day. The complexity of the infrastructure, as well as the reasoning behind choices that have been made in its …
Case Study: Understanding Kafka Architecture at LinkedIn Read More »
While we can use any type of database to persist the output of streaming pipelines, NoSQL is the most popular choice. Here are the list of market leading NoSQL databases. Overview of NoSQL Databases HBase – comes as part of Hadoop eco system with most of the distributions such as Cloudera, Hortonworks, MapR etc. Cassandra …
As part of this session we will see the overview of technologies used in building Streaming data pipelines. Also we will have deeper look into Spark Structured Streaming by developing solution for a simple problem. Overview of Streaming Technologies Ingestion Processing in real time Databases Visualization Frameworks for building streaming pipelines Spark Structured Streaming Basic …
Overview of Streaming Technologies and Spark Structured Streaming Read More »
As part of this topic we will see how we can develop programs to produce messages to Kafka Topic and consume messages from Kafka Topic using Scala as Programming language. Revision about Kafka Topic Understand producer as well as consumer APIs Create new project with KafkaWorkshop Define dependencies in build.sbt Create application.properties for passing zookeeper …
Let us see the steps involved in Kafka multi broker setup. We will be using simulated environment with 7 virtual machines. Pre-requisites for multi broker setup Setup Kafka and Zookeeper binaries Configure Zookeeper Zookeeper Commands Configure Kafka Brokers Validate Kafka Let us review some of the details related to one of the nodes being used …
