[ Overview of JSON¶ Let us get an overview of JSON. JSON stands for JavaScript Object Notation. It is group of Key Value Pairs. Keys are unique and values need not be unique. Both Keys as well as Values can be of any type. Strings should be enclosed in double quotes. Here is an example …
[ Processing JSON Data¶ As part of this section or module we will get an overview about processing JSON data. We will primarily focus on using json or pandas to process data in JSON format. Overview of JSON JSON Data Types Create JSON String Process JSON String Single JSON document in files Multiple JSON documents …
[ Execute Values for Batch Inserts¶ In [2]: %run 02_function_get_database_connection.ipynb In [3]: %load_ext sql In [4]: %env DATABASE_URL=postgresql://itversity_sms_user:itversity@pg.itversity.com:5432/itversity_sms_db env: DATABASE_URL=postgresql://itversity_sms_user:itversity@pg.itversity.com:5432/itversity_sms_db In [5]: %%sql DROP TABLE IF EXISTS courses Done. Out[5]: [] In [6]: %%sql CREATE TABLE courses ( course_id SERIAL PRIMARY KEY, course_name VARCHAR(60), course_author VARCHAR(40), course_status VARCHAR(9) CHECK (course_status IN (‘published’, ‘inactive’, ‘draft’)), course_published_dt DATE ) * postgresql://itversity_sms_user:***@pg.itversity.com:5432/itversity_sms_db Done. …
[ Best Practices – Batch Loading¶ Let us go through some of the best practices to perform batch load. We should minimize the number of connections to database. We should avoid executing queries using hard coded values. Rather, we should prefer bind variables. Too much committing is bad as commit incurs overhead. If we have …
[ Batch Loading of Data¶ Let us understand how we should take care of loading data in batches. We will perform load using multiple approaches to understand which one is better. Approach 1: Insert and commit each record. Whenever there is a commit in database, there is considerable amount of overhead. Approach 2: Insert one …
[ Reading Data from File¶ Let us read both orders as well as order_items data set from files into Pandas Data Frame. Both the files does not have header and hence we need to pass the schema while creating data frames. In [1]: # !ls -ltr /data/retail_db/orders/part-00000 In [2]: # !head /data/retail_db/orders/part-00000 In [3]: import pandas as pd …
