Repository navigation
The user_data doesn't belong exclusively to the user #1261
Description
Using user_data to pass data between handlers, I noticed that Crawlee also uses the dict for internally storing some stuff, such as label. This has surprised me. It's user data, right? Intuitively, I thought I can drop stuff to user_data, get it out in the other handler (well, it's not a dict anymore, so I need to do dict(context.request.user_data), and go on. But then I noticed it contains stuff which I didn't store there.
I guess the recommended way to use this is to always create a 'namespace' for my stuff, for example instead of:
Request.from_url(item.select_one(".item-link")["href"], label="article", user_data=data)I should do
Request.from_url(item.select_one(".item-link")["href"], label="article", user_data={"actual_user_data": data})Then I can't do
data = dict(context.request.user_data)but must do
data = context.request.user_data["actual_user_data"]Also it's not a dict, it's JsonSerializable[Unknown] and even dict(context.request.user_data["actual_user_data"]) gives a flood of type errors.
Why it's called user data when it's not mine? Why do I have to come up with a namespace for my user data? Why so much type handling? To me this feels like a design flaw, but perhaps it's intentional? Not sure if this is a bug or feature.
This will be a breaking change.
Activity
janbuchar commentedon Jun 23, 2025
Hi Honza, the problem is, it's user data from the perspective of the Apify platform, so "the user" means "crawlee and whoever uses crawlee". I agree that this is not at all what a user would expect, especially if they don't use the Apify platform. This arrangement is mostly historical baggage, I guess that at some point, we might be able to keep crawlee's user data and actual user data separated in the crawler code and only smush them together when sending the Request to the Platform.
Regarding the type issues, I think you (or your IDE) might be a bit confused.
context.request.user_datashould be of typedict[str, JsonSerializable]. If you work withcontext.request.user_data["actual_user_data"]in a different call to a request handler than the one in which you set the key, you don't really have any guarantee that what you find there actually has the type you expected (at least from the perspective of the type checker). So your options are either to cast it (and tell the type checker to shut up because you know better) or to validate it using a Pydantic model.Let us know if there is any specific code with type errors that we can help troubleshoot.
honzajavorek commentedon Jun 23, 2025
Thanks, understood!
I wrote a few scrapers on top of Crawlee and although I've been using Crawlee in the past and I've been through the drill, I made the mistake and had to fix my code, which made me think: Why the interface isn't more intuitive?
Also I might be too used to Scrapy's
cb_kwargs, which removes friction (as in passing the user data as keyword arguments to the handler), whereas in Crawlee I got bitten by an extra layer of friction which I didn't see a reason for. I very well understand the historical baggage reasoning, and I'm sending compassion your way.Regarding the types, you're right I cannot assume anything about the data I'm getting, neither the type checker can, so that's probably right.