As part of this section, we will see how to set up Kafka and HBase and explore some key concepts of both services. Kafka Overview Setup Parcels and Add Kafka Service Validate Kafka HBase Overview Configure HBase Validate HBase Cluster Topology We are setting up the cluster on 7+1 nodes. We start with 7 nodes …
As Hue is setup successfully, let us see an overview of Hue. We will setup an admin account, then add user and then run Hive query using Hue Web Interface. Setup admin account Once Hue installation is completed you can open the Hue UI from <installed-host-public-ip>:8888 Since we have not opened the firewall for 8888 …
Let us validate Sqoop by running sample job. We typically use Sqoop to get data from remote RDBMS databases into HDFS and vice versa. Sqoop uses JDBC to connect to remote databases. Hence on the gateway node, where we typically run Sqoop import/export commands, we need to have JDBC jar file. JDBC Jar is based …
Let us see how we can validate Pig job on our cluster. Pig uses HDFS for File System and Map Reduce to process the data. Ensure you have data to validate (in our case we have data in local file system /home/itversity/data). Let us copy data to HDFS. Create directory /user/itversity/data Copy whole directory on …
Here we will see how to run the default Map Reduce job using Oozie We can check the status of Oozie Server by running this command – oozie admin -oozie http://bigdataserver-3:11000/oozie -status Oozie have several sub-commands for different purposes – job, admin etc Create directory oozie_demo under home directory – /home/itversity Copy the oozie example provided …
Let us review property files as well as important properties for all 4 services – Oozie, Pig, Sqoop and Hue. Property Files – Standard Locations Oozie – /etc/oozie/conf Sqoop – /etc/sqoop/conf Pig – /etc/pig/conf Hue – /etc/hue/conf However, with Cloudera there will be only templates actual run time property files will be under /var/run/cloudera-scm-agent/process With …
