Let us see how we can add Kafka to the cluster using Cloudera Manager. We will also see how to review components, properties as well as log files with respect to Kafka.
- Go to Hosts -> Parcels
- If you want to add any other version of Kafka
- Click on Configuration
- Add a “Remote Parcel Repository URLs” for Apache Kafka 1.4.0 with the URL http://archive.cloudera.com/kafka/redhat/7/x86_64/kafka/
- Download the Kafka Parcel, Distribute and Activate.
- Add Service “Kafka” and select the hosts for brokers
- Hosts – bigdataserver-5,6 and 7
- Gateway – bigdataserver-1
- Review the properties and Click on Finish
- Kafka service should start but it may fail.
- Then go to Kafka Configuration Broker Heap Size value
- Change the configuration property – broker_max_heap_size from 50 Mb by default and change to 1 Gb.
- Go to Kafka -> Instances -> Restart Kafka Brokers
- Make sure to click on Kafka using Cloudera Manager and review summary.
- We can validate Kafka components by running ps -fu kafka
- We can review properties files either by using Cloudera Manager or CLI
- We can troubleshoot any issues using logs in Cloudera Manager or log files under /var/log/kafka.