Develop myReduceByKey Function¶ Develop a function by name myReduceByKey which takes a collection of tuples and a function as arguments. Each element in the collection should have exactly 2 attributes. Function should do the following: Iterate through the collection of tuples. Group the data by first element in the collection of tuples and apply the …
Validate myReduce Function¶ Let us validate myReduce Function. Compute order revenue for a given order id using order_items. We will use myFilter to filter for the order items for the given order id. Use myMap to extract order_item_subtotal. We will also convert data type of order_item_subtotal to float. We can now compute order_revenue using myReduce …
Develop myReduce Function¶ Develop a function by name myReduce which takes a collection and a function as arguments. The myReduce function should do the following: Iterate through elements Perform aggregation operation using the argument passed. Argument should have necessary arithmetic logic. Return the aggregated result. In [1]: l = [1, 4, 6, 2, 5] In [2]: l[1:] …
Validate myMap Function¶ Let us validate the function by using some realistic examples. In [1]: %run 04_develop_myMap_function.ipynb In [2]: myMap Out[2]: <function __main__.myMap(c, f)> Use orders and extract order_dates. Also apply set and get only unique dates. In [3]: orders_path = “/data/retail_db/orders/part-00000” orders = open(orders_path). \ read(). \ splitlines() In [4]: orders[:10] Out[4]: [‘1,2013-07-25 00:00:00.0,11599,CLOSED’, ‘2,2013-07-25 00:00:00.0,256,PENDING_PAYMENT’, ‘3,2013-07-25 00:00:00.0,12111,COMPLETE’, …
Develop myMap Function¶ Develop a function by name myMap which takes a collection and a function as arguments. myMap function should do the following: Iterate through elements. Apply the transformation logic using the argument passed. Append the transformed record to the list. Return the collection with all the elements which are transformed based on the …
Validate myFilter Function¶ Use the same examples which were used before as part of Processing Collections using loops. In [1]: %run 02_develop_myFilter_function.ipynb In [2]: myFilter Out[2]: <function __main__.myFilter(c, f)> Read orders data In [3]: orders_path = “/data/retail_db/orders/part-00000” orders = open(orders_path). \ read(). \ splitlines() In [4]: orders[:10] Out[4]: [‘1,2013-07-25 00:00:00.0,11599,CLOSED’, ‘2,2013-07-25 00:00:00.0,256,PENDING_PAYMENT’, ‘3,2013-07-25 00:00:00.0,12111,COMPLETE’, ‘4,2013-07-25 00:00:00.0,8827,CLOSED’, ‘5,2013-07-25 00:00:00.0,11318,COMPLETE’, ‘6,2013-07-25 …
