Skip to content

[SYSTEMDS-3972] Merge PR Differential privacy for aggregates - #2617

Closed
ywcb00 wants to merge 55 commits into
apache:mainfrom
ywcb00:pr-2539
Closed

ywcb00 wants to merge 55 commits into
apache:mainfrom
ywcb00:pr-2539

Conversation

@ywcb00

@ywcb00 ywcb00 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Intermediate PR for merging #2539.

(If anybody knows a better way to verify the CI tests when merging larger PRs with multiple changes that were made during the merge, please inform me.)

Maya Anderson added 28 commits September 18, 2026 15:10
…ransformation matrix T internally, returning T %*% X with noise fused into a single matrix multiply.
Lets a DML script declare its session-wide differential-privacy budget once at the top, instead of always falling back to the hardcoded default.
Resolved entirely at compile time: epsilon/delta must be literals, validated in BuiltinFunctionExpression and stored on DMLProgram during HOP construction, then read by ExecutionContext.getDPBudgetAccountant().
Four federated workers simulated on localhost, a logistic regression FedAvg loop in DML where the coordinator applies dp_gaussian to the aggregated gradient, a sweep over ε ∈ {0.5, 1, 4, 8} plus a non-private baseline, and a matplotlib accuracy-vs-ε plot saved as a PNG.

Add clip_norm (default 4.0) as a script parameter. Inside the private == 1 branch, each row's gradient contribution is clipped to L2-norm less than clip_norm.
…ntical dp_laplace or dp_gaussian calls are never be merged into one execution.
Flip Builtins.DP_LAPLACE/DP_GAUSSIAN to parameterized=true so the parser
builds a ParameterizedBuiltinFunctionExpression for these calls instead of
a positional BuiltinFunctionExpression. This lets them reuse the existing
varParams-based parsing, instead of hand-unpacking expr/expr2/expr3 by position in
DMLTranslator and re-deriving parameter names from argument order in
BuiltinFunctionExpression. Laplace and Gaussian validation is merged into
one validateDpMechanism(), varying only on whether 'delta' is required,
since the two mechanisms differ by exactly that one optional parameter.
Maya Anderson and others added 19 commits September 18, 2026 15:10
…nCPInstruction, owning parse validation, processInstruction(), and the lineage-refusal — all DP instruction-level concerns now live in one file
…hich defines two global constants

	define these constants as environment variables instead
	remove unnecessary comments and shorten method headers
@codecov

codecov Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.64516% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.80%. Comparing base (b42b641) to head (5e67549).

Files with missing lines Patch % Lines
...untime/instructions/cp/DPBuiltinCPInstruction.java 57.14% 5 Missing and 4 partials ⚠️
.../org/apache/sysds/hops/ParameterizedBuiltinOp.java 52.94% 1 Missing and 7 partials ⚠️
...apache/sysds/parser/BuiltinFunctionExpression.java 52.94% 4 Missing and 4 partials ⚠️
...he/sysds/runtime/instructions/cp/DPBuiltinOps.java 89.04% 5 Missing and 3 partials ⚠️
...parser/ParameterizedBuiltinFunctionExpression.java 75.86% 4 Missing and 3 partials ⚠️
...ntime/controlprogram/context/ExecutionContext.java 50.00% 0 Missing and 3 partials ⚠️
...in/java/org/apache/sysds/parser/DMLTranslator.java 77.77% 1 Missing and 1 partial ⚠️
...e/sysds/runtime/privacy/dp/DPBudgetAccountant.java 95.12% 2 Missing ⚠️
...ructions/cp/ParameterizedBuiltinCPInstruction.java 80.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##               main    #2617       +/-   ##
=============================================
- Coverage     74.94%   71.80%    -3.15%     
- Complexity        0    50705    +50705     
=============================================
  Files           436     2078     +1642     
  Lines         25237   222402   +197165     
  Branches          0    38306    +38306     
=============================================
+ Hits          18915   159685   +140770     
- Misses         6322    51630    +45308     
- Partials          0    11087    +11087     
Flag Coverage Δ
java 71.40% <80.64%> (?)
python 74.92% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

… parsing tests

fix(test/functions/privacy/dp/DPBuiltinDMLTest.java): use dml script files for test execution

feat(test/scripts/functions/privacy/dp/**): create dml test scripts
…oo fine-grained comments

	shorten method headers
…java test workflow

refactor(test/functions/privacy/dp/DPBuiltinDMLTest.java): move test file to the parent directory
…a warning that the differential privacy features are still experimental
@ywcb00 ywcb00 changed the title Merge PR Differential privacy for aggregates [SYSTEMDS-3972] Merge PR Differential privacy for aggregates Sep 20, 2026
@ywcb00 ywcb00 closed this in 4ebbdae Sep 20, 2026
@github-project-automation github-project-automation Bot moved this from In Progress to Done in SystemDS PR Queue Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant