Uncategorized

Command to import order_items table using compress sqoop import –connect jdbc:mysql://ms.itversity.com:3306/ retail_db –username retail_user –password itversity –table order_items –warehouse-dir /user/training/sqoop_import/retail_db –delete-target-dir –compress Command to import table as compressed file using a specific compression codec sqoop import –connect jdbc:mysql://ms.itversity.com:3306/retail_db –username retail_user –password itversity –table order_items –warehouse-dir /user/training/sqoop_import/retail_db –delete-target-dir –compress –compression-codec org.apache.hadoop.io.compress.SnappyCodec

Command to delete directory if exists in local file system: rm -rf order_items To copy data from hdfs to local file system: hadoop fs -get /user/training/sqoop_import/retail_db/order_items . To extract schema from file in json format using avro-tools: avro-tools getschema part-m-00000.avro To print first few records from avro file in json format: avro-tools tojson part-m-00000.avro | …

179.Validating avro Files using “avro-tools” Read More »

To save the imported table as avro file format: sqoop import –connect jdbc:mysql –username retail_user –password itversity –table order_items –warehouse-dir /user/training/sqoop_import/retail_db –delete-target-dir –as-avrodatafile To validate imported files: hadoop fs -ls /user/training/sqoop_import/retail_db/order_items

To get the list of files in current directory: ls -ltr To view a file: view part-m-00000 To validate order_items table: sqoop eval –connect jdbc:mysql://ms.itversity.com:3306/retail_db –username retail_user –password itversity -e “DESCRIBE order_items”

Command to import table and customize the number of mappers: sqoop import –connect jdbc:mysql://ms.itversity.com:3306/retail_db –username retail_user –password itversity –table order_items –warehouse-dir /user/training/sqoop_import/retail_db –delete-target-dir –num-mappers 8 To validate: hadoop fs -ls /user/training/sqoop_import/retail_db/order_items

To redirect log messages to other files sqoop import –connect jdbc:mysql://ms.itversity.com:3306/retail_db –username retail_user –password itversity –table order_items –warehouse-dir /user/training/sqoop_import/retail_db –delete-target-dir 1>import.out 2>import.err To view the import.out file: view import.out To view the import.err file: view import.err Command to get status of the job: mapred job -status job_id