Transforming data using mapPartitionsWithIndex

  • MapPartitionsWithIndex takes associated index as well as iterator.

val nysePartitions= nyse.mapPartitionsWithIndex( (index, iter) =>{
iter.map(nyseRec=>{
val a = nyseRec.split(",")  ((a(1).toInt,a(0)),a(6).toLong)})
})

  • This creates an RDD and gives same result as map and map partitions.