Repository navigation
Variable self._is_first_run in ListRedisScheduleSource doesn't work #89
Description
Activity
Yeah. I should fix this comment. Because as it turned out, you should fetch previous schedules every time. Because otherwise you loose them completely.
Reacted by Lryan@s3rius The ListRedisScheduleSource currently runs a full SCAN across the entire Redis keyspace on every update_interval to discover past time keys. When Redis holds a large number of keys
(millions+), this becomes a significant bottleneck — it causes heavy CPU load on both Redis and the scheduler process, and the latency scales linearly with total key count regardless of how many
schedule keys actually exist.I've submitted a fix in #121 that eliminates the SCAN entirely by maintaining a {prefix}:time_index sorted set that tracks time keys as they're
created. On each poll, it uses ZRANGEBYSCORE to look up only the relevant past keys — an O(log N + M) operation where M is just the number of matching time keys, completely independent of total
Redis key count.Key changes:
- No more SCAN — past time key discovery uses ZRANGEBYSCORE on the sorted set index
- Automatic bookkeeping — add_schedule adds to the index, delete_schedule triggers lazy cleanup of stale entries (rate-limited, with a 5-minute safety window to avoid race conditions)
- Zero-downtime migration — a populate_time_index=True flag lets existing users backfill the index with a one-time SCAN on startup, then disable it for all future runs
- Fixed a bug — in an extremely rare case, get_schedules could fetch current minute schedules twice if both get_schedules():current_time and _get_previous_time_schedules():minute_before point to the same minute.
All existing tests pass and 10 new tests cover the index lifecycle, cleanup behavior, and migration path. Would appreciate a review when you get a chance — happy to adjust anything.
taskiq-redis/taskiq_redis/list_schedule_source.py
Line 193 in 513b77e
According to the comments, this condition should be
Otherwise, the variable
self._is_first_runis not used at all.