Allow cleaning up of networks stuck in Implementing state by Pearl1594 · Pull Request #12409 · apache/cloudstack

Pearl1594 · 2026-01-12T16:53:49Z

Description

This PR fixes: #12249
This PR allows cleanup of Networks that have been stuck in Implementing state for more than the job.expire.minutes setting and has no async jobs associated.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Created a network
Created a VM in that network and cleaned up the VM
Set network (manually in DB) to Implementing state
Observe logs (temporarily placed one of the trace logs in the code as debug)

...
Found 1 networks in Implementing state, checking for orphaned networks
2026-01-12 16:43:58,432 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:[ctx-abe618b0]) (logid:5ee73b3f) Network 204 in Implementing state is only 9 minutes old (threshold: 10 minutes), skipping cleanup
..
 Found 1 networks in Implementing state, checking for orphaned networks
2026-01-12 16:44:08,434 WARN  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:[ctx-5012868f]) (logid:8c11718f) Found orphaned network 204 in Implementing state without async job. Network created: 2026-01-12T16:34:05.000+0000, age: 10 minutes, expiration threshold: 10 minutes. Transitioning to Shutdown state.

...
<img width="1621" height="147" alt="image" src="https://github.com/user-attachments/assets/8aadf16e-c022-4502-b88f-7c090a18e5f3" />

Moves network to Shutdown state and from there the networks can be deleted.

How did you try to break this feature and the system with this change?

codecov · 2026-01-12T17:01:59Z

Codecov Report

❌ Patch coverage is 0% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.65%. Comparing base (4348386) to head (d6fe913).
⚠️ Report is 99 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...stack/framework/jobs/impl/AsyncJobManagerImpl.java	0.00%	57 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.20   #12409      +/-   ##
============================================
+ Coverage     16.21%   16.65%   +0.43%     
- Complexity    13353    14136     +783     
============================================
  Files          5657     5660       +3     
  Lines        498647   520073   +21426     
  Branches      60533    69103    +8570     
============================================
+ Hits          80863    86603    +5740     
- Misses       408771   424021   +15250     
- Partials       9013     9449     +436

Flag	Coverage Δ
uitests	`5.00% <ø> (+0.99%)`	⬆️
unittests	`17.47% <0.00%> (+0.40%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

DaanHoogland

one nitpit, I would disect cleanupOrphanedNetworks() further as well.

framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

Pearl1594 · 2026-01-28T17:28:37Z

@blueorangutan package

blueorangutan · 2026-01-28T17:30:05Z

@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-01-28T18:23:23Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16600

nvazquez · 2026-01-29T11:43:31Z

@blueorangutan test

blueorangutan · 2026-01-29T11:44:04Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-01-30T02:44:27Z

[SF] Trillian test result (tid-15335)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 51086 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12409-t15335-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

sudo87

clgtm

sonarqubecloud · 2026-02-04T14:58:33Z

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube Cloud

DaanHoogland

clgtm

sureshanaparti · 2026-02-05T08:15:16Z

framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

+            sc.addAnd("removed", SearchCriteria.Op.NULL);
+            List<NetworkVO> implementingNetworks = networkDao.search(sc, null);
+
+            if (implementingNetworks == null || implementingNetworks.isEmpty()) {


Suggested change

if (implementingNetworks == null || implementingNetworks.isEmpty()) {

if (CollectionUtils.isEmpty(implementingNetworks)) {

sureshanaparti · 2026-02-05T08:17:41Z

framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

+                List<AsyncJobVO> jobs = _jobDao.findInstancePendingAsyncJobs("Network", network.getAccountId());
+                boolean hasActiveJob = false;
+                for (AsyncJobVO job : jobs) {
+                    if (job.getInstanceId() != null && job.getInstanceId().equals(network.getId())) {


can pass these checks to jobDao and find the active job?

Allow cleaning up of networks stuck in Implementing state

bfbb8bb

Pearl1594 requested review from DaanHoogland, kiranchavala and weizhouapache January 12, 2026 16:54

DaanHoogland reviewed Jan 13, 2026

View reviewed changes

framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java Outdated Show resolved Hide resolved

Pearl1594 added this to the 4.20.3 milestone Jan 28, 2026

nvazquez self-assigned this Jan 29, 2026

nvazquez marked this pull request as ready for review January 29, 2026 11:43

sudo87 approved these changes Jan 30, 2026

View reviewed changes

DaanHoogland linked an issue Feb 4, 2026 that may be closed by this pull request

Unable to delete or cleanup a network whose state is struck in Implementing #12249

Open

extract code

d6fe913

DaanHoogland approved these changes Feb 5, 2026

View reviewed changes

sureshanaparti reviewed Feb 5, 2026

View reviewed changes

	if (implementingNetworks == null \|\| implementingNetworks.isEmpty()) {
	if (CollectionUtils.isEmpty(implementingNetworks)) {

Conversation

Pearl1594 commented Jan 12, 2026

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pearl1594 commented Jan 28, 2026

Uh oh!

blueorangutan commented Jan 28, 2026

Uh oh!

blueorangutan commented Jan 28, 2026

Uh oh!

nvazquez commented Jan 29, 2026

Uh oh!

blueorangutan commented Jan 29, 2026

Uh oh!

blueorangutan commented Jan 30, 2026

Uh oh!

sudo87 left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Feb 4, 2026

Quality Gate failed

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

sureshanaparti Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov bot commented Jan 12, 2026 •

edited

Loading