It is better to have Linux based environment for setting up Big Data Clusters. It provides us below capabilities. Ability to connect to remote servers using ssh Copy files using scp or rsync Enable proxy such as sshuttle to access web applications running on the servers behind firewall with out opening up ports to the public using ssh authentication. Setup Process …
As part of this section, we will see how to set up Pig and Oozie components and some of the key concepts related to each service. Setup Oozie, Pig, Sqoop and Hue Review Important Properties Schedule an Oozie workflow Run Pig Job Validate Sqoop Overview of Hue Cluster Topology We are setting up the cluster …
Configure Hadoop Ecosystem components – Oozie, Pig, Sqoop and Hue Read More »
As part of this section we will talk about setting up necessary tools to create virtual machines on Google Cloud Platform (GCP) Setup Ubuntu using Windows Subsystem Sign up for GCP Create template for Big Data Server Provision Servers for Big Data Cluster Review Concepts Setting up gcloud Setup ansible on first server Format JBOD …
[ Setup Database Client Libraries¶ Let us understand how to install Database Connector to connect to Postgres Database and run queries. We will also try to connect to database and submit a query as part of the validation. You can install Postgres Python connector using python3 -m pip install psycopg2 or python3 -m pip install …
Running OS Commands¶ Let us understand how to run OS commands using Python using libraries such as os and subprocess.* Python provides several libraries which can be used to run OS commands. os and subprocess are most popular ones. We can import the libraries such as os and subprocess to start using them. There are …
Using CASE and WHEN¶ At times we might have to select values from multiple columns conditionally. We can use CASE and WHEN for that. Let us implement this conditional logic to come up with derived order_status. If order_status is COMPLETE or CLOSED, set COMPLETED If order_status have PENDING in it, then we will say PENDING …
