Overview of Retail Data¶ We will be using data from hypothetical retail application. Let us get an overview about the data sets. We typically setup data under /data/retail_db There are 6 folders by following names. departments categories products customers orders order_items In [1]: !ls -ltr /data/retail_db total 20128 drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 …
File Paths and Names¶ Let us quickly review about file paths and names. Typically, we will have folder and file name associated while processing a file. A file might be part of a directory structure with multiple levels. All the directories in the path are typically separated by /. The last part of path is …
Folders and Files¶ Let us quickly recap details about folders and files, especially using Linux. You need to be comfortable with the following. Differentiating Files and Folders. Keep in mind that Folders and Directories means the samething. Understanding Absolute or Fully Qualified Path. Understanding Relative Path. Understanding File or Folder permissions. In [1]: # Listing files …
Overview of File IO¶ Let us get an overview of File IO. Files are typically saved in hard drive and we need to perform file i/o to read the data from the files into memory as objects. Here are the commom operations related to files. Read the data from the files into objects (typically collections). …
Basics of File I/O using Python¶ As part of this section or module we will go through some of the basics related to File I/O using Python as programming language. Overview of File I/O Folders and Files File Paths and Names Overview of Retail Data Read Text File into String Write String to Text File …
Limitations of Map Reduce Libraries¶ Here are some of the limitations of using Map Reduce Libraries. We cannot refer attributes with names directly. Functions are scattered and lack consistency. Readability and maintainability are addressed using libraries such as Pandas. Libraries such as PySpark takes care of scalability. {note} We use the approach of loops or …
