Durga Gadiraju

Durga Gadiraju

Install and Configure Kafka and HBase

As part of this section, we will see how to set up Kafka and HBase and explore some key concepts of both services.

  • Kafka Overview
  • Setup Parcels and Add Kafka Service
  • Validate Kafka
  • HBase Overview
  • Configure HBase
  • Validate HBase

Cluster Topology

We are setting up the cluster on 7+1 nodes. We start with 7 nodes and then we will add one more node later.

  • Gateway(s) and Management Serviceh2
    • bigdataserver-1 – Hue Server
  • Masters
    • bigdataserver-2
      • Zookeeper
      • Active/Standby Namenode
      • HBase Master
    • bigdataserver-3
      • Zookeeper
      • Active/Standby Namenode
      • Active/Standby Resource Manager
      • Impala State Store
      • Oozie Server
      • HBase Master
    • bigdataserver-4
      • Zookeeper
      • Active/Standby Resource Manager
      • Job History Server
      • Spark History Server
      • Hive Server and Hive Metastore
      • Impala Catalog
      • HBase Master
  • Slaves or Worker Nodes
    • bigdataserver-5 – Datanode, Node Manager, Impala Daemon, Region Server, Kafka Broker
    • bigdataserver-6 – Datanode, Node Manager, Impala Daemon, Region Server, Kafka Broker
    • bigdataserver-7 – Datanode, Node Manager, Impala Daemon, Region Server, Kafka Broker

Kafka is typically set up as an external cluster to Big Data Clusters. However, in our environment, we will be setting up 3 broker Kafka Cluster on worker nodes.

Share this post