Quick start

This page gets you from installation to a running crawler in a few minutes. To build a complete scraper step by step, follow the Introduction.

Choose your crawler

Crawlee's main crawler classes are BeautifulSoupCrawler, ParselCrawler, and PlaywrightCrawler. All crawlers share the same interface, so you can switch between them without rewriting your handlers.

Minimum Python version

Crawlee requires Python 3.10 or higher.

BeautifulSoupCrawler

The BeautifulSoupCrawler is a plain HTTP crawler that parses HTML using the well-known BeautifulSoup library. It fetches pages with an HTTP client that mimics a browser. It's fast and light, but it can't render JavaScript.

ParselCrawler

The ParselCrawler is similar to the BeautifulSoupCrawler but uses the Parsel library for HTML parsing. Parsel extracts data from HTML with CSS selectors and XPath, and it's the selector library Scrapy uses. Like the BeautifulSoupCrawler, the ParselCrawler can't render JavaScript.

PlaywrightCrawler

The PlaywrightCrawler uses a headless browser controlled by the Playwright library. It can drive Chromium, Firefox, and WebKit. Choose it when the target pages render their content with JavaScript.

Installation

Crawlee is available as the crawlee package on PyPI. The package includes the core functionality, and additional features are available as optional extras to keep dependencies small.

To install Crawlee with all extras using pip, run:

python -m pip install 'crawlee[all]'

Verify that Crawlee is successfully installed:

python -c 'import crawlee; print(crawlee.__version__)'

If you plan to use the PlaywrightCrawler, also install the Playwright browser binaries:

playwright install

For detailed installation instructions, see Setting up.

Crawling

Run the following example to perform a recursive crawl of the Crawlee website using the selected crawler.

BeautifulSoupCrawler

Run on

import asyncio

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext


async def main() -> None:
    # BeautifulSoupCrawler crawls the web using HTTP requests
    # and parses HTML using the BeautifulSoup library.
    crawler = BeautifulSoupCrawler(max_requests_per_crawl=10)

    # Define a request handler to process each crawled page
    # and attach it to the crawler using a decorator.
    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')
        # Extract relevant data from the page context.
        data = {
            'url': context.request.url,
            'title': context.soup.title.string if context.soup.title else None,
        }
        # Store the extracted data.
        await context.push_data(data)
        # Extract links from the current page and add them to the crawling queue.
        await context.enqueue_links()

    # Add first URL to the queue and start the crawl.
    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

ParselCrawler

Run on

import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext


async def main() -> None:
    # ParselCrawler crawls the web using HTTP requests
    # and parses HTML using the Parsel library.
    crawler = ParselCrawler(max_requests_per_crawl=10)

    # Define a request handler to process each crawled page
    # and attach it to the crawler using a decorator.
    @crawler.router.default_handler
    async def request_handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')
        # Extract relevant data from the page context.
        data = {
            'url': context.request.url,
            'title': context.selector.xpath('//title/text()').get(),
        }
        # Store the extracted data.
        await context.push_data(data)
        # Extract links from the current page and add them to the crawling queue.
        await context.enqueue_links()

    # Add first URL to the queue and start the crawl.
    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

PlaywrightCrawler

Run on

import asyncio

from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    # PlaywrightCrawler crawls the web using a headless browser
    # controlled by the Playwright library.
    crawler = PlaywrightCrawler()

    # Define a request handler to process each crawled page
    # and attach it to the crawler using a decorator.
    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f'Processing {context.request.url} ...')
        # Extract relevant data from the page context.
        data = {
            'url': context.request.url,
            'title': await context.page.title(),
        }
        # Store the extracted data.
        await context.push_data(data)
        # Extract links from the current page and add them to the crawling queue.
        await context.enqueue_links()

    # Add first URL to the queue and start the crawl.
    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

When you run the example, the crawler logs its progress to your terminal.

Running headful browser

By default, browsers controlled by Playwright run in headless mode, without a visible window. During development, it helps to run the browser in headful mode and watch what it does. You can also switch from the default Chromium browser to Firefox or WebKit.

import asyncio

from crawlee.crawlers import PlaywrightCrawler


async def main() -> None:
    crawler = PlaywrightCrawler(
        # Run with a visible browser window.
        headless=False,
        # Switch to the Firefox browser.
        browser_type='firefox',
    )

    # ...


if __name__ == '__main__':
    asyncio.run(main())

When you run the example, a browser window opens and navigates through the Crawlee website.

Results

By default, Crawlee stores data in the ./storage directory within your current working directory. The crawl results are saved as JSON files under ./storage/datasets/default/.

To view the results, use the cat command:

cat ./storage/datasets/default/000000001.json

The JSON file contains data similar to the following:

{
    "url": "https://crawlee.dev/",
    "title": "Crawlee · Build reliable crawlers. Fast. | Crawlee"
}

tip

To change the storage directory, set the CRAWLEE_STORAGE_DIR environment variable to your preferred path.

Next steps

• To build a complete scraper for a real website step by step, follow the Introduction.
• To learn how crawlers, requests, storages, and other parts of Crawlee work, read the Concepts.
• For solutions to specific tasks, see the Guides.