Repository navigation
Metrics for deleted CRs may lead to OOM when scraping #1803
Description
Bug Report
What did you do?
Request to /metrics endpoint in Quarkus (using JOSDK w/Quarkus extension) resulted in an OOM error. Further research revealed that JOSDK reconcile metrics accumulate in the Micrometer registry and are not removed for CRs no longer present in the system.
What did you expect to see?
Expect metrics for deleted CRs be removed from the registry after a configurable amount of time, with a default of -1/never.
What did you see instead? Under which circumstances?
Metrics for deleted CRs are present until the process ends
Environment
Kubernetes cluster type:
$ Mention java-operator-sdk version from pom.xml file
JOSDK: 3.2.4 (via quarkus-operator-sdk 4.0.7) on Quarkus 2.13.7
$ java -version
11.0.x
$ kubectl version
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4+a34b9e9", GitCommit:"b6d1f054747e9886f61dd85316deac3415e2726f", GitTreeState:"clean", BuildDate:"2023-01-10T15:55:28Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Possible Solution
Clear/remove metrics for deleted CRs after a configurable amount of time, defaulting to "never" for compatibility.
Additional context
Metadata
Metadata
Assignees
Labels
No labels
Activity
metacosm commentedon Mar 8, 2023
Would upgrading to a more recent version of the SDK be a possibility as, at least on first approximation, we would only fix this on the
mainbranch, i.e. for JOSDK 4.2 / 4.3 (Quarkus extension 5.0/5.1)?MikeEdgar commentedon Mar 9, 2023
I think it will probably be a while before we can upgrade again. We only just migrated to 3.2.4 (extension v4.0.7) For now, we're looking at using a no-op implementation of
Metrics.feat: remove CR meters when they are deleted (after a delay)
feat: remove CR meters when they are deleted (after a delay)
feat: remove CR meters when they are deleted (after a delay)
feat: remove CR meters when they are deleted (after a delay)
feat: remove CR meters when they are deleted (after a delay) (#1805)
feat: remove CR meters when they are deleted (after a delay) (#1805)
feat: remove CR meters when they are deleted (after a delay) (#1805)
feat: remove CR meters when they are deleted (after a delay) (#1805)