Filtering Data using filter¶ Let us go through the details about filter. We can use filter on top of iterable to return new iterable with all the elements satisfying the condition. It takes filter logic and iterable as arguments. We can pass filter logic either as regular function or lambda function. filter returns a special …
Preparing Data Sets¶ We will be primarily using orders and order_items data set to understand about manipulating collections. orders is available at path /data/retail_db/orders/part-00000 order_items is available at path /data/retail_db/order_items/part-00000 orders – columns order_id – it is of type integer and unique order_date – it can be considered as string order_customer_id – it is of …
Preparing Data Sets for Understanding Map-Reduce Libraries Read More »
Validate myJoin Function¶ Let us perform a task to validate myJoin. In this case we will be joining order_id and order_date from orders with order_id and order_item_subtotal from order_items. Read orders data set from file. Create list of tuples using order_id and order_date with order_id as first element. We can use myMap function developed earlier …
Develop myJoin Function¶ Develop a function by name myJoin which takes two collections of tuples as arguments. Each element in each of the collection should have exactly 2 attributes. Function should do the following: Build the dict for the first collection. Iterate through the second collection and look up into the dict. Return a collection …
Validate myReduceByKey Function¶ Let us perform few tasks to validate myReduceKey Function. In [1]: %run 04_develop_myMap_function.ipynb In [2]: %run 08_develop_myReduceByKey_function.ipynb Use the function to get the count by date from orders. In [3]: orders_path = “/data/retail_db/orders/part-00000” In [4]: orders = open(orders_path). \ read(). \ splitlines() In [5]: orders[:10] Out[5]: [‘1,2013-07-25 00:00:00.0,11599,CLOSED’, ‘2,2013-07-25 00:00:00.0,256,PENDING_PAYMENT’, ‘3,2013-07-25 00:00:00.0,12111,COMPLETE’, ‘4,2013-07-25 00:00:00.0,8827,CLOSED’, ‘5,2013-07-25 00:00:00.0,11318,COMPLETE’, ‘6,2013-07-25 …
Develop myReduceByKey Function¶ Develop a function by name myReduceByKey which takes a collection of tuples and a function as arguments. Each element in the collection should have exactly 2 attributes. Function should do the following: Iterate through the collection of tuples. Group the data by first element in the collection of tuples and apply the …
