Data Ingestion with Python Cookbook offers a practical approach to designing and implementing data ingestion pipelines. It presents real-world examples with the most widely recognized open source tools on the market to answer commonly asked questions and overcome challenges.

You'll be introduced to designing and working with or without data schemas, as well as creating monitored pipelines with Airflow and data observability principles, all while following industry best practices. The book also addresses challenges associated with reading different data sources and data formats. As you progress through the book, you'll gain a broader understanding of error logging best practices, troubleshooting techniques, data orchestration, monitoring, and storing logs for further consultation.

By the end of the book, you'll have a fully automated set that enables you to start ingesting and monitoring your data pipeline effortlessly, facilitating seamless integration with subsequent stages of the ETL process.

What you will learn

Implement data observability using monitoring tools

Automate your data ingestion pipeline

Read analytical and partitioned data, whether schema or non-schema based

Debug and prevent data loss through efficient data monitoring and logging

Establish data access policies using a data governance framework

Construct a data orchestration framework to improve data quality

Who this book is for

This book is for data engineers and data enthusiasts seeking a comprehensive understanding of the data ingestion process using popular tools in the open source community. For more advanced learners, this book takes on the theoretical pillars of data governance while providing practical examples of real-world scenarios commonly encountered by data engineers.

Table of Contents

Introduction to Data Ingestion

Principals of Data Access – Accessing your Data

Data Discovery – Understanding Our Data Before Ingesting It

Reading CSV and JSON Files and Solving Problems

Ingesting Data from Structured and Unstructured Databases

Using PySpark with De?ned and Non-De?ned Schemas

Ingesting Analytical Data

Designing Monitored Data Workflows

Putting Everything Together with Air?ow

Logging and Monitoring Your Data Ingest in Airflow

Automating Your Data Ingestion Pipelines

Using Data Observability for Debugging, Error Handling, and Preventing Downtime