_job_start_time records the populating host's local time, with no zone #1584
Copy link
Copy link
Open
Milestone
Description
populate() stores _job_start_time as naive local time on whichever host runs it: start_time = time.time() (autopopulate.py:665), then written as datetime.datetime.fromtimestamp(start_time) (autopopulate.py:742). Two hosts in different time zones write different values for the same instant, and nothing in the row records which zone applies. A populate() run from a laptop in US Central and one from a worker in UTC leave start times five hours apart for computations that ran at the same moment.
The other timestamps the framework writes use a known clock:
_prov.timeis UTC:datetime.datetime.now(datetime.timezone.utc).isoformat()(provenance.py:103).- The job table's
completed_timeis the server'sCURRENT_TIMESTAMP(jobs.py:499,529).
A reader cannot place _job_start_time on either of those scales unless every host that populates the schema runs in the same zone.
Proposal
- Take
_job_start_timefrom the server's clock, as the job table already does forcompleted_time(jobs.py:498-499) andscheduled_time(jobs.py:388). Compute it in theUPDATEthat already writes the hidden columns (autopopulate.py:874), as the server's current time minus_job_duration. That adds no round trip, and every host that populates the schema then writes on one clock, whatever zone the host runs in. - The server expression must be statement time, not transaction time:
NOW(3)on MySQL, andstatement_timestamp()on PostgreSQL, whereCURRENT_TIMESTAMPis the start of the transaction. It belongs in the adapter, besidecurrent_user_expr(). - Keep measuring
_job_durationon the worker. It is a difference of two readings on one clock, so neither zone nor skew affects it. TheUPDATEruns after the commit that_job_durationalready includes, so the derived start is late by the one-way latency of that statement, which is the same error a separateSELECTat start would carry. - State in
job-metadata.md(datajoint-docs) that_job_start_timeuses the server's clock from the release that ships the change, and that rows written earlier carry the populating host's local time. Those rows cannot be corrected after the fact, because the zone was never recorded.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity