This book covers the long awaited Scrapy v 1.0 that empowers you to extract useful data from virtually any source with very little effort. It starts off by explaining the fundamentals of Scrapy framework, followed by a thorough description of how to extract data from any source, clean it up, shape it as per your requirement using Python and 3rd party APIs. Next you will be familiarised with the process of storing the scrapped data in databases as well as search engines and performing real time analytics on them with Spark Streaming. By the end of this book, you will perfect the art of scarping data for your applications with ease

What you will learn

Understand HTML pages and write XPath to extract the data you need

Write Scrapy spiders with simple Python and do web crawls

Push your data into any database, search engine or analytics system

Configure your spider to download files, images and use proxies

Create efficient pipelines that shape data in precisely the form you want

Use Twisted Asynchronous API to process hundreds of items concurrently

Make your crawler super-fast by learning how to tune Scrapy's performance

Perform large scale distributed crawls with scrapyd and scrapinghub

About the Author

Dimitrios Kouzis-Loukas

has over fifteen years experience as a topnotch software developer. He uses his acquired knowledge and expertise to teach a wide range of audiences how to write great software, as well.

He studied and mastered several disciplines, including mathematics, physics, and microelectronics. His thorough understanding of these subjects helped him raise his standards beyond the scope of "pragmatic solutions." He knows that true solutions should be as certain as the laws of physics, as robust as ECC memories, and as universal as mathematics.

Dimitrios now develops distributed, low-latency, highly-availability systems using the latest datacenter technologies. He is language agnostic, yet has a slight preference for Python, C++, and Java. A firm believer in open source software and hardware, he hopes that his contributions will benefit individual communities as well as all of humanity.

Table of Contents

Introducing Scrapy

Understanding HTML and XPath

Basic Crawling

From Scrapy to a Mobile App

Quick Spider Recipes

Deploying to Scrapinghub

Configuration and Management

Programming Scrapy

Pipeline Recipes

Understanding Scrapy's Performance

Distributed Crawling with Scrapyd and Real-Time Analytics

Installing and troubleshooting prerequisite software

Також купити книгу Learning Scrapy, Dimitris Kouzis - Loukas Ви можете по посиланню