Running OS Commands¶
Let us understand how to run OS commands using Python using libraries such as os and subprocess.* Python provides several libraries which can be used to run OS commands. os and subprocess are most popular ones.
- We can import the libraries such as
osandsubprocessto start using them. - There are bunch of commands to create directories, change ownership, change permission, run general system commands etc.
oslibrary is extensively used to read environment variables at run time of the application. It is used to pass keys and credentials to work with databases, external applications etc.- Typically keys and credentials should not be part of the source code.
subprocesscan be used to run the commands and also to process the output.
In [1]:
import os
- Get current working directory.
In [2]:
os.getcwd()
Out[2]:
'/home/itversity/itversity-material/01-python-and-sql/11_basic_programming_constructs'
- Read environment variables
In [3]:
os.environ.get('PATH')
Out[3]:
'/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
In [4]:
os.environ.get('USER')
In [5]:
os.environ.get('HOME')
Out[5]:
'/home/itversity'
In [6]:
%%sh
env|grep -v JUPYTER|grep -v TOKEN|grep -v PASSWORD
MPLBACKEND=module://matplotlib_inline.backend_inline HOSTNAME=itvdflab SHLVL=1 PYTHON_PIP_VERSION=21.2.4 HOME=/home/itversity GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568 PAGER=cat _=/home/itversity/.local/bin/jupyter PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/2caf84b14febcda8077e59e9b8a6ef9a680aa392/public/get-pip.py TERM=xterm-color PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=C.UTF-8 SHELL=/bin/bash PYTHON_VERSION=3.8.12 PYTHON_SETUPTOOLS_VERSION=57.5.0 GIT_PAGER=cat PWD=/home/itversity/itversity-material/01-python-and-sql/11_basic_programming_constructs CLICOLOR=1 PYTHON_GET_PIP_SHA256=7c5239cea323cadae36083079a5ee6b2b3d56f25762a0c060d2867b89e5e06c5 JPY_PARENT_PID=7 PYDEVD_USE_FRAME_EVAL=NO
In [7]:
os.environ.get?
Signature: os.environ.get(key, default=None) Docstring: D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None. File: /usr/local/lib/python3.8/_collections_abc.py Type: method
In [8]:
os.environ.get('PASSWORD', 'Passwords should be confidential')
Out[8]:
'Passwords should be confidential'
- Run
ls -ltrcommand to get list of files in the current directory.
In [9]:
import subprocess
In [10]:
%%sh
ls -ltr
total 140 -rw-rw-r-- 1 itversity itversity 4915 Mar 8 02:04 15_overview_of_unhandled_exceptions.ipynb -rw-rw-r-- 1 itversity itversity 4076 Mar 8 02:04 14_else_and_finally_in_exceptions.ipynb -rw-rw-r-- 1 itversity itversity 5323 Mar 8 02:04 13_overview_of_exception_handling.ipynb -rw-rw-r-- 1 itversity itversity 2934 Mar 8 02:04 12_eval_and_exec.ipynb -rw-rw-r-- 1 itversity itversity 3945 Mar 8 02:04 11_exercises_basic_programming_constructs.ipynb -rw-rw-r-- 1 itversity itversity 16782 Mar 8 02:04 10_running_os_commands.ipynb -rw-rw-r-- 1 itversity itversity 9804 Mar 8 02:04 09_all_about_for_loops.ipynb -rw-rw-r-- 1 itversity itversity 930 Mar 14 16:23 01_basic_programming_constructs.ipynb -rw-rw-r-- 1 itversity itversity 31861 Mar 14 16:23 02_getting_help.ipynb -rw-rw-r-- 1 itversity itversity 4078 Mar 14 16:24 03_variables_and_objects.ipynb -rw-rw-r-- 1 itversity itversity 1370 Mar 14 16:24 04_data_types_commonly_used.ipynb -rw-rw-r-- 1 itversity itversity 5043 Mar 14 16:24 05_overview_of_boolean_type.ipynb -rw-rw-r-- 1 itversity itversity 10332 Mar 14 16:25 06_operators_in_python.ipynb -rw-rw-r-- 1 itversity itversity 6181 Mar 14 16:26 07_comments_and_doc_strings.ipynb -rw-rw-r-- 1 itversity itversity 7922 Mar 14 16:29 08_conditionals.ipynb
In [11]:
output = subprocess.check_call(['ls', '-ltr'])
total 144 -rw-rw-r-- 1 itversity itversity 4915 Mar 8 02:04 15_overview_of_unhandled_exceptions.ipynb -rw-rw-r-- 1 itversity itversity 4076 Mar 8 02:04 14_else_and_finally_in_exceptions.ipynb -rw-rw-r-- 1 itversity itversity 5323 Mar 8 02:04 13_overview_of_exception_handling.ipynb -rw-rw-r-- 1 itversity itversity 2934 Mar 8 02:04 12_eval_and_exec.ipynb -rw-rw-r-- 1 itversity itversity 3945 Mar 8 02:04 11_exercises_basic_programming_constructs.ipynb -rw-rw-r-- 1 itversity itversity 16782 Mar 8 02:04 10_running_os_commands.ipynb -rw-rw-r-- 1 itversity itversity 930 Mar 14 16:23 01_basic_programming_constructs.ipynb -rw-rw-r-- 1 itversity itversity 31861 Mar 14 16:23 02_getting_help.ipynb -rw-rw-r-- 1 itversity itversity 4078 Mar 14 16:24 03_variables_and_objects.ipynb -rw-rw-r-- 1 itversity itversity 1370 Mar 14 16:24 04_data_types_commonly_used.ipynb -rw-rw-r-- 1 itversity itversity 5043 Mar 14 16:24 05_overview_of_boolean_type.ipynb -rw-rw-r-- 1 itversity itversity 10332 Mar 14 16:25 06_operators_in_python.ipynb -rw-rw-r-- 1 itversity itversity 6181 Mar 14 16:26 07_comments_and_doc_strings.ipynb -rw-rw-r-- 1 itversity itversity 7922 Mar 14 16:29 08_conditionals.ipynb -rw-rw-r-- 1 itversity itversity 12307 Mar 14 16:30 09_all_about_for_loops.ipynb
In [12]:
output
Out[12]:
0
In [13]:
output = subprocess.check_output(['ls', '-ltr'])
In [14]:
output # output is of type binary
Out[14]:
b'total 144\n-rw-rw-r-- 1 itversity itversity 4915 Mar 8 02:04 15_overview_of_unhandled_exceptions.ipynb\n-rw-rw-r-- 1 itversity itversity 4076 Mar 8 02:04 14_else_and_finally_in_exceptions.ipynb\n-rw-rw-r-- 1 itversity itversity 5323 Mar 8 02:04 13_overview_of_exception_handling.ipynb\n-rw-rw-r-- 1 itversity itversity 2934 Mar 8 02:04 12_eval_and_exec.ipynb\n-rw-rw-r-- 1 itversity itversity 3945 Mar 8 02:04 11_exercises_basic_programming_constructs.ipynb\n-rw-rw-r-- 1 itversity itversity 16782 Mar 8 02:04 10_running_os_commands.ipynb\n-rw-rw-r-- 1 itversity itversity 930 Mar 14 16:23 01_basic_programming_constructs.ipynb\n-rw-rw-r-- 1 itversity itversity 31861 Mar 14 16:23 02_getting_help.ipynb\n-rw-rw-r-- 1 itversity itversity 4078 Mar 14 16:24 03_variables_and_objects.ipynb\n-rw-rw-r-- 1 itversity itversity 1370 Mar 14 16:24 04_data_types_commonly_used.ipynb\n-rw-rw-r-- 1 itversity itversity 5043 Mar 14 16:24 05_overview_of_boolean_type.ipynb\n-rw-rw-r-- 1 itversity itversity 10332 Mar 14 16:25 06_operators_in_python.ipynb\n-rw-rw-r-- 1 itversity itversity 6181 Mar 14 16:26 07_comments_and_doc_strings.ipynb\n-rw-rw-r-- 1 itversity itversity 7922 Mar 14 16:29 08_conditionals.ipynb\n-rw-rw-r-- 1 itversity itversity 12307 Mar 14 16:30 09_all_about_for_loops.ipynb\n'
In [15]:
type(output)
Out[15]:
bytes
In [16]:
output.decode('utf-8') # converts to string of type utf-8
Out[16]:
'total 144\n-rw-rw-r-- 1 itversity itversity 4915 Mar 8 02:04 15_overview_of_unhandled_exceptions.ipynb\n-rw-rw-r-- 1 itversity itversity 4076 Mar 8 02:04 14_else_and_finally_in_exceptions.ipynb\n-rw-rw-r-- 1 itversity itversity 5323 Mar 8 02:04 13_overview_of_exception_handling.ipynb\n-rw-rw-r-- 1 itversity itversity 2934 Mar 8 02:04 12_eval_and_exec.ipynb\n-rw-rw-r-- 1 itversity itversity 3945 Mar 8 02:04 11_exercises_basic_programming_constructs.ipynb\n-rw-rw-r-- 1 itversity itversity 16782 Mar 8 02:04 10_running_os_commands.ipynb\n-rw-rw-r-- 1 itversity itversity 930 Mar 14 16:23 01_basic_programming_constructs.ipynb\n-rw-rw-r-- 1 itversity itversity 31861 Mar 14 16:23 02_getting_help.ipynb\n-rw-rw-r-- 1 itversity itversity 4078 Mar 14 16:24 03_variables_and_objects.ipynb\n-rw-rw-r-- 1 itversity itversity 1370 Mar 14 16:24 04_data_types_commonly_used.ipynb\n-rw-rw-r-- 1 itversity itversity 5043 Mar 14 16:24 05_overview_of_boolean_type.ipynb\n-rw-rw-r-- 1 itversity itversity 10332 Mar 14 16:25 06_operators_in_python.ipynb\n-rw-rw-r-- 1 itversity itversity 6181 Mar 14 16:26 07_comments_and_doc_strings.ipynb\n-rw-rw-r-- 1 itversity itversity 7922 Mar 14 16:29 08_conditionals.ipynb\n-rw-rw-r-- 1 itversity itversity 12307 Mar 14 16:30 09_all_about_for_loops.ipynb\n'
In [17]:
type(output.decode('utf-8'))
Out[17]:
str
{note}
Let us convert string into list of strings. Once it is broken into list of strings we can process the data as per our requirements either by using Map Reduce libraries or Pandas based libraries.In [18]:
output.decode('utf-8').splitlines()
Out[18]:
['total 144', '-rw-rw-r-- 1 itversity itversity 4915 Mar 8 02:04 15_overview_of_unhandled_exceptions.ipynb', '-rw-rw-r-- 1 itversity itversity 4076 Mar 8 02:04 14_else_and_finally_in_exceptions.ipynb', '-rw-rw-r-- 1 itversity itversity 5323 Mar 8 02:04 13_overview_of_exception_handling.ipynb', '-rw-rw-r-- 1 itversity itversity 2934 Mar 8 02:04 12_eval_and_exec.ipynb', '-rw-rw-r-- 1 itversity itversity 3945 Mar 8 02:04 11_exercises_basic_programming_constructs.ipynb', '-rw-rw-r-- 1 itversity itversity 16782 Mar 8 02:04 10_running_os_commands.ipynb', '-rw-rw-r-- 1 itversity itversity 930 Mar 14 16:23 01_basic_programming_constructs.ipynb', '-rw-rw-r-- 1 itversity itversity 31861 Mar 14 16:23 02_getting_help.ipynb', '-rw-rw-r-- 1 itversity itversity 4078 Mar 14 16:24 03_variables_and_objects.ipynb', '-rw-rw-r-- 1 itversity itversity 1370 Mar 14 16:24 04_data_types_commonly_used.ipynb', '-rw-rw-r-- 1 itversity itversity 5043 Mar 14 16:24 05_overview_of_boolean_type.ipynb', '-rw-rw-r-- 1 itversity itversity 10332 Mar 14 16:25 06_operators_in_python.ipynb', '-rw-rw-r-- 1 itversity itversity 6181 Mar 14 16:26 07_comments_and_doc_strings.ipynb', '-rw-rw-r-- 1 itversity itversity 7922 Mar 14 16:29 08_conditionals.ipynb', '-rw-rw-r-- 1 itversity itversity 12307 Mar 14 16:30 09_all_about_for_loops.ipynb']
In [19]:
type(output.decode('utf-8').splitlines())
Out[19]:
list
In [20]:
# splitlines is the function available on string type
# It converts string with line breaks into list of strings
for rec in output.decode('utf-8').splitlines():
print(rec)
total 144 -rw-rw-r-- 1 itversity itversity 4915 Mar 8 02:04 15_overview_of_unhandled_exceptions.ipynb -rw-rw-r-- 1 itversity itversity 4076 Mar 8 02:04 14_else_and_finally_in_exceptions.ipynb -rw-rw-r-- 1 itversity itversity 5323 Mar 8 02:04 13_overview_of_exception_handling.ipynb -rw-rw-r-- 1 itversity itversity 2934 Mar 8 02:04 12_eval_and_exec.ipynb -rw-rw-r-- 1 itversity itversity 3945 Mar 8 02:04 11_exercises_basic_programming_constructs.ipynb -rw-rw-r-- 1 itversity itversity 16782 Mar 8 02:04 10_running_os_commands.ipynb -rw-rw-r-- 1 itversity itversity 930 Mar 14 16:23 01_basic_programming_constructs.ipynb -rw-rw-r-- 1 itversity itversity 31861 Mar 14 16:23 02_getting_help.ipynb -rw-rw-r-- 1 itversity itversity 4078 Mar 14 16:24 03_variables_and_objects.ipynb -rw-rw-r-- 1 itversity itversity 1370 Mar 14 16:24 04_data_types_commonly_used.ipynb -rw-rw-r-- 1 itversity itversity 5043 Mar 14 16:24 05_overview_of_boolean_type.ipynb -rw-rw-r-- 1 itversity itversity 10332 Mar 14 16:25 06_operators_in_python.ipynb -rw-rw-r-- 1 itversity itversity 6181 Mar 14 16:26 07_comments_and_doc_strings.ipynb -rw-rw-r-- 1 itversity itversity 7922 Mar 14 16:29 08_conditionals.ipynb -rw-rw-r-- 1 itversity itversity 12307 Mar 14 16:30 09_all_about_for_loops.ipynb