Skip to content

Fix Clojure v4 strict load error handling - #1775

Open
ThunderRonin wants to merge 2 commits into
dimitri:mainfrom
ThunderRonin:fix/v4-on-error-stop
Open

ThunderRonin wants to merge 2 commits into
dimitri:mainfrom
ThunderRonin:fix/v4-on-error-stop

Conversation

@ThunderRonin

Copy link
Copy Markdown

Problem

In the Clojure v4 path, WITH on error stop could still enter row rejection, commit the valid part of a failed batch, and exit successfully. MSSQL JDBC row-advance errors were also treated as EOF, while column-read errors became SQL NULL.

Change

  • propagate dynamic COPY settings into table and partition executor tasks with bound-fn
  • roll back and propagate PostgreSQL COPY errors in strict mode
  • preserve existing batch retry/rejection behavior in resume mode
  • propagate MSSQL ResultSet.next and getString errors
  • stop queued table COPY work after a strict failure is observed
  • let the writer terminate after draining a full queue when its reader fails
  • preserve the original COPY error if rollback itself fails
  • propagate table-worker failures after active sibling workers finish
  • add the focused tests to the Makefile and CI unit-test lists

This remains batch-transactional rather than globally transactional: batches committed before a later failure are not rolled back.

Reproduction and validation

A two-row MSSQL fixture (1, not-an-integer) loaded into a PostgreSQL integer column reproduced the bug:

Build Exit Rows committed from failed batch
Current v4 0 1
This change 1 0

Additional isolated validation copied 57 MSSQL tables / 10,115,173 rows through the patched v4 path. Independent all-column reconciliation matched all 57 per-table counts and two domain-separated SHA-256 multiset digests, with no operational timestamp/text discrepancies.

Local checks:

  • clojure -M:cljfmt check src test
  • clojure -T:build uber
  • make test-unit — 136 tests, 485 assertions, 0 failures/errors

Closes #1774.

Propagate dynamic load settings into executor workers, abort strict COPY failures after rollback, and surface MSSQL JDBC read errors instead of treating them as EOF or NULL. Add focused unit coverage for strict versus resume behavior and source read failures.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clojure v4: on error stop can silently commit a partial load

1 participant