Skip to content

Add progress logging (monitor mode) for scraping #770

Open

Description

@vdusek

❗ Note: This is a starting point for discussion, not a final decision. The team should review and adjust before any implementation begins.❗

A feedback from @honzajavorek:

I do think that Crawlee should log some "progress" information about requests made or - especially - items scraped. It's so weird to run the program and then just look at the program as if it hanged, waiting if something happens or not. E.g. Scrapy logs how many items per minute I scraped, which I personally find super useful.

Activity

  1. added
    t-toolingIssues with this label are in the ownership of the tooling team.
    on Dec 2, 2024
  2. janbuchar commented on Dec 9, 2024

    @janbuchar
    Collaborator

    See apify/crawlee#2692 in JS crawlee - it strives to implement something that fulfills the same use case IMO

  3. changed the title [-]Add progress logging for scraping[/-] [+]Add progress logging (monitor mode) for scraping[/+] on Feb 17, 2026
  4. added this to the 2.0 milestone on Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request.t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions