You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Wrap the discovery client with memory.NewMemCacheClient() to cache
OpenAPI v3 schema responses. This prevents repeated fetching and
unmarshaling of schemas during ClusterExtensionRevision reconciliation.
The boxcutter machinery uses the discovery client to fetch OpenAPI
schemas for resource validation and comparison. Without caching, these
schemas are fetched and parsed on every reconciliation, leading to
excessive memory allocations.
Testing shows significant improvements:
- Peak memory usage reduced by 16.9% (8.4 MB)
- Memory growth reduced by 29.3% (10.5 MB)
- OpenAPI-related allocations reduced by 73% (~9.5 MB)
- Eliminated repeated schema unmarshaling operations
- Extended test duration by 8% before OOM
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
⚡ Optimize memory usage with cache transforms and reduced copying
This commit implements several memory optimizations that reduce peak
memory usage during e2e tests by ~7.8% (6.57 MB):
1. Strip managed fields and large annotations from cached objects
- Add DefaultTransform function to cache that removes managed fields
- Remove kubectl.kubernetes.io/last-applied-configuration annotations
- Applied to all objects before storing in informer caches
2. Optimize label copying in revision generation
- Replace maps.Clone with direct allocation and copy
- Pre-allocate maps with correct capacity
- Reduces unnecessary DeepCopy operations by 37%
3. Strip metadata from revision objects
- Remove managed fields and large annotations from objects
- Applied in both Helm and plain manifest processing paths
Memory impact (measured via pprof during test-experimental-e2e):
- Peak memory: 84.58 MB → 78.01 MB (-6.57 MB, -7.8%)
- DeepCopyJSONValue: 17.50 MB → 11 MB (-6.5 MB, -37%)
- Sustained 7-14K reduction per snapshot throughout test execution
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
⚡ Add slice pre-allocation optimizations
Optimize slice allocations in boxcutter to reduce memory overhead:
1. Pre-allocate trimmedPrevious with ClusterExtensionRevisionPreviousLimit capacity
- Avoids reallocation as the slice grows
2. Smart pre-allocation in splitManifestDocuments
- Estimates document count based on line count
- Reduces allocations when processing large helm manifests
- Minimum capacity of 4 for small bundles
These micro-optimizations reduce GC pressure during revision processing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
⚡ Apply memory optimizations to catalogd
Apply the same memory optimization patterns used in operator-controller
to catalogd for consistent memory management across the codebase:
1. Add cache transform function to strip managed fields and annotations
- Removes managed fields from all cached objects
- Strips kubectl.kubernetes.io/last-applied-configuration annotations
- Applied to all catalogd informer caches
2. Pre-allocate slices with correct capacity
- localdir.go: Pre-allocate metaChans with len(storeMetaFuncs)
- garbage_collector.go: Pre-allocate removed slice with len(cacheDirEntries)
- Reduces allocations and GC pressure during catalog operations
These optimizations follow the same patterns that reduced operator-controller
memory usage by 38% and should provide similar benefits for catalogd.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
♻️ Refactor: Share cache transform function between components
Extract the stripManagedFieldsAndAnnotations function to a shared
utility package to eliminate code duplication between operator-controller
and catalogd.
Changes:
- Created internal/shared/util/cache/transform.go with
StripManagedFieldsAndAnnotations function
- Updated cmd/operator-controller/main.go to use shared implementation
- Updated cmd/catalogd/main.go to use shared implementation
- Removed duplicate function definitions (46 lines of duplication removed)
This improves maintainability by having a single source of truth for
the cache transform logic used across both components.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: CONTRIBUTING.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -153,7 +153,7 @@ Please follow this style to make the operator-controller project easier to revie
153
153
154
154
Our goal is to minimize disruption by requiring the lowest possible Go language version. This means avoiding updaties to the go version specified in the project's `go.mod` file (and other locations).
155
155
156
-
There is a GitHub PR CI job named `go-verdiff` that will inform a PR author if the Go language version has been updated. It is not a required test, but failures should prompt authors and reviewers to have a discussion with the community about the Go language version change.
156
+
There is a GitHub PR CI job named `go-verdiff` that will inform a PR author if the Go language version has been updated. It is not a required test, but failures should prompt authors and reviewers to have a discussion with the community about the Go language version change.
157
157
158
158
There may be ways to avoid a Go language version change by using not-the-most-recent versions of dependencies. We do acknowledge that CVE fixes might require a specific dependency version that may have updated to a newer version of the Go language.
Copy file name to clipboardExpand all lines: docs/concepts/permission-model.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,7 +18,7 @@ To understand the permission model, lets see the scope of the the service accoun
18
18
19
19
##### Example:
20
20
21
-
Lets consider deployment of the ArgoCD operator. The ClusterExtension ClusterResource specifies a service account as part of its spec, usually denoted as the ClusterExtension installer service account.
21
+
Lets consider deployment of the ArgoCD operator. The ClusterExtension ClusterResource specifies a service account as part of its spec, usually denoted as the ClusterExtension installer service account.
22
22
The ArgoCD operator specifies the `argocd-operator-controller-manager`[service account](https://github.com/argoproj-labs/argocd-operator/blob/da6b8a7e68f71920de9545152714b9066990fc4b/deploy/olm-catalog/argocd-operator/0.6.0/argocd-operator.v0.6.0.clusterserviceversion.yaml#L1124) with necessary RBAC for the bundle resources and OLMv1 creates it as part of this extension bundle deployment.
23
23
24
24
The extension bundle CSV contains the [permissions](https://github.com/argoproj-labs/argocd-operator/blob/da6b8a7e68f71920de9545152714b9066990fc4b/deploy/olm-catalog/argocd-operator/0.6.0/argocd-operator.v0.6.0.clusterserviceversion.yaml#L1091) and [cluster permissions](https://github.com/argoproj-labs/argocd-operator/blob/da6b8a7e68f71920de9545152714b9066990fc4b/deploy/olm-catalog/argocd-operator/0.6.0/argocd-operator.v0.6.0.clusterserviceversion.yaml#L872) allow the operator to manage and run the controller logic. These permissions are assigned to the `argocd-operator-controller-manager` service account when the operator bundle is deployed.
Copy file name to clipboardExpand all lines: docs/draft/api-reference/catalogd-webserver-metas-endpoint.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,9 +5,9 @@ a web server that serves catalog contents to clients via HTTP(S) endpoints.
5
5
6
6
The endpoints to retrieve information about installable clusterextentions can be composed from the `.status.urls.base` of a `ClusterCatalog` resource with the selected access API path.
7
7
8
-
Currently, there are two API endpoints:
8
+
Currently, there are two API endpoints:
9
9
10
-
1.`api/v1/all` endpoint that provides access to the FBC metadata in entirety.
10
+
1.`api/v1/all` endpoint that provides access to the FBC metadata in entirety.
11
11
12
12
As an example, to access the full FBC via the v1 API endpoint (indicated by path `api/v1/all`) where `.status.urls.base` is
13
13
@@ -18,7 +18,7 @@ As an example, to access the full FBC via the v1 API endpoint (indicated by path
18
18
19
19
the URL to access the service would be `https://catalogd-service.olmv1-system.svc/catalogs/operatorhubio/api/v1/all`
20
20
21
-
2. `api/v1/metas` endpoint that allows clients to retrieve filtered portions of the FBC.
21
+
2. `api/v1/metas` endpoint that allows clients to retrieve filtered portions of the FBC.
22
22
23
23
The metas endpoint accepts parameters which are one of the sub-types of the `Meta` [definition](https://github.com/operator-framework/operator-registry/blob/e15668c933c03e229b6c80025fdadb040ab834e0/alpha/declcfg/declcfg.go#L111-L114), following the pattern `/api/v1/metas?<parameter>[&<parameter>...]`.
Copy file name to clipboardExpand all lines: docs/draft/api-reference/network-policies.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
## Overview
4
4
5
-
OLMv1 uses [Kubernetes NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) to secure communication between components, restricting network traffic to only what's necessary for proper functionality.
5
+
OLMv1 uses [Kubernetes NetworkPolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) to secure communication between components, restricting network traffic to only what's necessary for proper functionality.
6
6
7
7
* The catalogd NetworkPolicy is implemented [here](https://github.com/operator-framework/operator-controller/blob/main/helm/olmv1/templates/networkpolicy/networkpolicy-olmv1-system-catalogd-controller-manager.yml).
8
8
* The operator-controller is implemented [here](https://github.com/operator-framework/operator-controller/blob/main/helm/olmv1/templates/networkpolicy/networkpolicy-olmv1-system-operator-controller-controller-manager.yml).
* Inspect logs: Check component logs for connection errors
90
90
91
-
For more comprehensive information on NetworkPolicy, see:
91
+
For more comprehensive information on NetworkPolicy, see:
92
92
93
93
- How NetworkPolicy is implemented with [network plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) via the Container Network Interface (CNI)
Copy file name to clipboardExpand all lines: docs/draft/howto/catalog-queries-metas-endpoint.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,9 +12,9 @@ Then you can query the catalog by using `curl` commands and the `jq` CLI tool to
12
12
By default, Catalogd is installed with TLS enabled for the catalog webserver.
13
13
The following examples will show this default behavior, but for simplicity's sake will ignore TLS verification in the curl commands using the `-k` flag.
14
14
15
-
!!! note
16
-
While using the `/api/v1/metas` endpoint shown in the below examples, it is important to note that the metas endpoint accepts parameters which are one of the sub-types of the `Meta`[definition](https://github.com/operator-framework/operator-registry/blob/e15668c933c03e229b6c80025fdadb040ab834e0/alpha/declcfg/declcfg.go#L111-L114), following the pattern `/api/v1/metas?<parameter>[&<parameter>...]`. e.g. `schema=<schema_name>&package=<package_name>`, `schema=<schema_name>&name=<name>`, and `package=<package_name>&name=<name>` are all valid parameter combinations. However `schema=<schema_name>&version=<version_string>` is not a valid parameter combination, since version is not a first class FBC meta field.
17
-
15
+
!!! note
16
+
While using the `/api/v1/metas` endpoint shown in the below examples, it is important to note that the metas endpoint accepts parameters which are one of the sub-types of the `Meta`[definition](https://github.com/operator-framework/operator-registry/blob/e15668c933c03e229b6c80025fdadb040ab834e0/alpha/declcfg/declcfg.go#L111-L114), following the pattern `/api/v1/metas?<parameter>[&<parameter>...]`. e.g. `schema=<schema_name>&package=<package_name>`, `schema=<schema_name>&name=<name>`, and `package=<package_name>&name=<name>` are all valid parameter combinations. However `schema=<schema_name>&version=<version_string>` is not a valid parameter combination, since version is not a first class FBC meta field.
17
+
18
18
You also need to port forward the catalog server service:
19
19
20
20
```terminal
@@ -51,7 +51,7 @@ Now you can use the `curl` command with `jq` to query catalogs that are installe
51
51
`<package_name>`
52
52
: Name of the package from the catalog you are querying.
53
53
54
-
Note: the `olm.package` schema blob does not have the `package` field set. In other words, to get all the blobs that belong to a package, along with the olm.package blob for that package, a combination of both of the above queries need to be used.
54
+
Note: the `olm.package` schema blob does not have the `package` field set. In other words, to get all the blobs that belong to a package, along with the olm.package blob for that package, a combination of both of the above queries need to be used.
0 commit comments