Sai

Preparing Data Sets¶ We will be primarily using orders and order_items data set to understand about manipulating collections. orders is available at path /data/retail_db/orders/part-00000 order_items is available at path /data/retail_db/order_items/part-00000 orders – columns order_id – it is of type integer and unique order_date – it can be considered as string order_customer_id – it is of …

Preparing Data Sets for Understanding Map-Reduce Libraries Read More »

Validate myJoin Function¶ Let us perform a task to validate myJoin. In this case we will be joining order_id and order_date from orders with order_id and order_item_subtotal from order_items. Read orders data set from file. Create list of tuples using order_id and order_date with order_id as first element. We can use myMap function developed earlier …

Validating MyJoin Functions Read More »

Develop myJoin Function¶ Develop a function by name myJoin which takes two collections of tuples as arguments. Each element in each of the collection should have exactly 2 attributes. Function should do the following: Build the dict for the first collection. Iterate through the second collection and look up into the dict. Return a collection …

Developing MyJoin Function Read More »

Validate myReduceByKey Function¶ Let us perform few tasks to validate myReduceKey Function. In [1]: %run 04_develop_myMap_function.ipynb In [2]: %run 08_develop_myReduceByKey_function.ipynb Use the function to get the count by date from orders. In [3]: orders_path = “/data/retail_db/orders/part-00000” In [4]: orders = open(orders_path). \ read(). \ splitlines() In [5]: orders[:10] Out[5]: [‘1,2013-07-25 00:00:00.0,11599,CLOSED’, ‘2,2013-07-25 00:00:00.0,256,PENDING_PAYMENT’, ‘3,2013-07-25 00:00:00.0,12111,COMPLETE’, ‘4,2013-07-25 00:00:00.0,8827,CLOSED’, ‘5,2013-07-25 00:00:00.0,11318,COMPLETE’, ‘6,2013-07-25 …

Validating MyReduceByKey Function Read More »

Develop myReduceByKey Function¶ Develop a function by name myReduceByKey which takes a collection of tuples and a function as arguments. Each element in the collection should have exactly 2 attributes. Function should do the following: Iterate through the collection of tuples. Group the data by first element in the collection of tuples and apply the …

Develop MyReduceByKey function Read More »

Validate myReduce Function¶ Let us validate myReduce Function. Compute order revenue for a given order id using order_items. We will use myFilter to filter for the order items for the given order id. Use myMap to extract order_item_subtotal. We will also convert data type of order_item_subtotal to float. We can now compute order_revenue using myReduce …

Validate MyReduce Function Read More »