Install and Configure Kafka and HBase

As part of this section, we will see how to set up Kafka and HBase and explore some key concepts of both services.

• Kafka Overview
• Setup Parcels and Add Kafka Service
• Validate Kafka
• HBase Overview
• Configure HBase
• Validate HBase

Cluster Topology

We are setting up the cluster on 7+1 nodes. We start with 7 nodes and then we will add one more node later.

• Gateway(s) and Management Serviceh2
  • bigdataserver-1 – Hue Server
• Masters
  • bigdataserver-2
    • Zookeeper
    • Active/Standby Namenode
    • HBase Master
  • bigdataserver-3
    • Zookeeper
    • Active/Standby Namenode
    • Active/Standby Resource Manager
    • Impala State Store
    • Oozie Server
    • HBase Master
  • bigdataserver-4
    • Zookeeper
    • Active/Standby Resource Manager
    • Job History Server
    • Spark History Server
    • Hive Server and Hive Metastore
    • Impala Catalog
    • HBase Master
• Slaves or Worker Nodes
  • bigdataserver-5 – Datanode, Node Manager, Impala Daemon, Region Server, Kafka Broker
  • bigdataserver-6 – Datanode, Node Manager, Impala Daemon, Region Server, Kafka Broker
  • bigdataserver-7 – Datanode, Node Manager, Impala Daemon, Region Server, Kafka Broker

Kafka is typically set up as an external cluster to Big Data Clusters. However, in our environment, we will be setting up 3 broker Kafka Cluster on worker nodes.