Managing Folders and Files using Python¶
Let us quickly recap details about folders and files, especially using Linux. You need to be comfortable with the following.
- Differentiating Files and Folders. Keep in mind that Folders and Directories means the samething.
- Understanding Absolute or Fully Qualified Path.
- Understanding Relative Path.
- Understanding File or Folder permissions.
We will first see how we can leverage subprocess to run Linux commands to manage files and then we will go through the above topics.
We will be running the following commands using Python subprocess module. That way, you will also pick up the relevance of subprocess as well.
# Listing files and folders under /data/retail_db
ls -ltr /data/retail_db
# Listing files in the present working directory
ls -ltr
# Listing files in the home directory. ~ represents home directory.
# You can find the ~ key on the left side of 1 in most of the keyboards.
ls -ltr ~
In [1]:
import subprocess
In [2]:
# Output is not reader friendly.
# Output is of type bytes
subprocess.check_output('ls -ltr /data/retail_db', shell=True)
Out[2]:
b'total 20128\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 categories\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 customers\n-rw-rw-r-- 1 itversity itversity 1748 Mar 8 02:04 create_db_tables_pg.sql\n-rw-rw-r-- 1 itversity itversity 10303297 Mar 8 02:04 create_db.sql\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 departments\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 order_items\n-rw-rw-r-- 1 itversity itversity 10297372 Mar 8 02:04 load_db_tables_pg.sql\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 orders\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 products\n'
In [3]:
# We can decode to string and apply string functions
# Now the output is of type string
subprocess.check_output('ls -ltr /data/retail_db', shell=True).decode('utf-8')
Out[3]:
'total 20128\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 categories\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 customers\n-rw-rw-r-- 1 itversity itversity 1748 Mar 8 02:04 create_db_tables_pg.sql\n-rw-rw-r-- 1 itversity itversity 10303297 Mar 8 02:04 create_db.sql\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 departments\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 order_items\n-rw-rw-r-- 1 itversity itversity 10297372 Mar 8 02:04 load_db_tables_pg.sql\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 orders\ndrwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 products\n'
In [4]:
# We can use splitlines to convert this big string into list of strings.
# splitlines will use new line character as the delimiter.
subprocess.check_output('ls -ltr /data/retail_db', shell=True).decode('utf-8').splitlines()
Out[4]:
['total 20128', 'drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 categories', 'drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 customers', '-rw-rw-r-- 1 itversity itversity 1748 Mar 8 02:04 create_db_tables_pg.sql', '-rw-rw-r-- 1 itversity itversity 10303297 Mar 8 02:04 create_db.sql', 'drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 departments', 'drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 order_items', '-rw-rw-r-- 1 itversity itversity 10297372 Mar 8 02:04 load_db_tables_pg.sql', 'drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 orders', 'drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 products']
In [5]:
# We can iterate through the list and print one item at a time.
# Now the output will be reader friendly
output = subprocess.check_output('ls -ltr /data/retail_db', shell=True).decode('utf-8').splitlines()
for line in output: print(line)
total 20128 drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 categories drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 customers -rw-rw-r-- 1 itversity itversity 1748 Mar 8 02:04 create_db_tables_pg.sql -rw-rw-r-- 1 itversity itversity 10303297 Mar 8 02:04 create_db.sql drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 departments drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 order_items -rw-rw-r-- 1 itversity itversity 10297372 Mar 8 02:04 load_db_tables_pg.sql drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 orders drwxrwxr-x 2 itversity itversity 24 Mar 8 02:04 products
In [6]:
subprocess.check_output('ls -ltr', shell=True).decode('utf-8').splitlines()
Out[6]:
['total 100', '-rw-rw-r-- 1 itversity itversity 4730 Mar 8 02:04 12_managing_files_and_folders_using_python.ipynb', '-rw-rw-r-- 1 itversity itversity 10177 Mar 8 02:04 11_write_strings_to_file_in_append_mode.ipynb', '-rw-rw-r-- 1 itversity itversity 5141 Mar 8 02:04 10_read_csv_into_list_of_strings.ipynb', '-rw-rw-r-- 1 itversity itversity 3584 Mar 8 02:04 09_overview_of_delimited_strings.ipynb', '-rw-rw-r-- 1 itversity itversity 1212 Mar 25 06:05 01_basics_of_file_io_using_python.ipynb', '-rw-rw-r-- 1 itversity itversity 1698 Mar 25 06:05 02_overview_of_file_io.ipynb', '-rw-rw-r-- 1 itversity itversity 9490 Mar 25 06:05 03_folders_and_files.ipynb', '-rw-rw-r-- 1 itversity itversity 3995 Mar 25 06:05 04_file_paths_and_names.ipynb', '-rw-rw-r-- 1 itversity itversity 6444 Mar 25 06:05 05_overview_of_retail_data.ipynb', '-rw-rw-r-- 1 itversity itversity 13941 Mar 25 06:05 06_read_text_file_into_string.ipynb', '-rw-rw-r-- 1 itversity itversity 7266 Mar 25 06:06 07_write_string_to_text_file.ipynb', 'drwxrwxr-x 2 itversity itversity 111 Mar 25 06:06 data', '-rw-rw-r-- 1 itversity itversity 10277 Mar 25 06:06 08_overview_of_modes_to_write_into_files.ipynb']
In [7]:
!pwd
/home/itversity/itversity-material/01-python-and-sql/20_basics_of_file_io_using_python
In [8]:
subprocess.check_output('ls -l', shell=True).decode('utf-8').splitlines()
Out[8]:
['total 100', '-rw-rw-r-- 1 itversity itversity 1212 Mar 25 06:05 01_basics_of_file_io_using_python.ipynb', '-rw-rw-r-- 1 itversity itversity 1698 Mar 25 06:05 02_overview_of_file_io.ipynb', '-rw-rw-r-- 1 itversity itversity 9490 Mar 25 06:05 03_folders_and_files.ipynb', '-rw-rw-r-- 1 itversity itversity 3995 Mar 25 06:05 04_file_paths_and_names.ipynb', '-rw-rw-r-- 1 itversity itversity 6444 Mar 25 06:05 05_overview_of_retail_data.ipynb', '-rw-rw-r-- 1 itversity itversity 13941 Mar 25 06:05 06_read_text_file_into_string.ipynb', '-rw-rw-r-- 1 itversity itversity 7266 Mar 25 06:06 07_write_string_to_text_file.ipynb', '-rw-rw-r-- 1 itversity itversity 10277 Mar 25 06:06 08_overview_of_modes_to_write_into_files.ipynb', '-rw-rw-r-- 1 itversity itversity 3584 Mar 8 02:04 09_overview_of_delimited_strings.ipynb', '-rw-rw-r-- 1 itversity itversity 5141 Mar 8 02:04 10_read_csv_into_list_of_strings.ipynb', '-rw-rw-r-- 1 itversity itversity 10177 Mar 8 02:04 11_write_strings_to_file_in_append_mode.ipynb', '-rw-rw-r-- 1 itversity itversity 4730 Mar 8 02:04 12_managing_files_and_folders_using_python.ipynb', 'drwxrwxr-x 2 itversity itversity 111 Mar 25 06:06 data']
In [9]:
subprocess.check_output('ls -ltr ~', shell=True).decode('utf-8').splitlines()
Out[9]:
['total 0', 'drwxrwxr-x 5 itversity itversity 91 Mar 8 02:04 itversity-material']
In [10]:
subprocess.check_output('ls -ltr data', shell=True).decode('utf-8').splitlines()
Out[10]:
['total 20', '-rw-r--r-- 1 itversity itversity 41 Mar 25 06:04 sample_data.txt', '-rw-r--r-- 1 itversity itversity 27 Mar 25 06:04 overwrite.txt', '-rw-r--r-- 1 itversity itversity 54 Mar 25 06:04 append.txt', '-rw-r--r-- 1 itversity itversity 27 Mar 25 06:04 new_file.txt', '-rw-r--r-- 1 itversity itversity 59 Mar 25 06:06 departments.txt']