Skip to content

feat(RELEASE-2158): Support self-hosted Quay#3148

Open
querti wants to merge 1 commit intoconforma:mainfrom
querti:self-hosted-quay
Open

feat(RELEASE-2158): Support self-hosted Quay#3148
querti wants to merge 1 commit intoconforma:mainfrom
querti:self-hosted-quay

Conversation

@querti
Copy link

@querti querti commented Mar 2, 2026

Verify Conforma task should support images that are in a self-hosted Quay instance. This instance has a self-signed CA cert which is implicitly not trusted by 3rd party tools. This can be fixed by mounting the CA bundle to /etc/ssl/certs/, making it trusted. Mount to /mnt/trusted-ca is added to be consistent with other tasks in the pipeline.

@querti querti force-pushed the self-hosted-quay branch from 1a35e17 to c68a985 Compare March 2, 2026 14:44
querti added a commit to querti/release-service-catalog that referenced this pull request Mar 2, 2026
Add a new e2e test pipeline that checks if push-to-external-registry
works with self-hosted Quay. Unlike other e2e tests which use the
stage cluster, this one uses ephemeral Kind cluster. The reason is
Quay is deployed to the cluster and is used for testing.

Two changes were made to the push-to-external-registry pipeline to
ensure its compatibility with self-hosted Quay:
- Mount the ca-bundle.crt from trusted-ca configmap to
  /etc/ssl/certs/. This ensures that the self-signed CA cert of the
  internal Quay instance is known to the 3rd party tools like skopeo
  and cosign. Without it, the tools would fail with a TLS error.
- Update regexes to support a Quay URL with a port number. The
  regexes assume that only one ":" will be present in the url,
  which may be wrong. Update them to account for hostnames with
  port number.

Several other changes were made to support this functionality:
- Mounting the CA bundle in Conforma task[1]
- Support self-hosted Quay deployment in the Kind cluster[2]
- Add init-quay task and skip-quay parameter[3]
- Add a new Go e2e test for self-hosted quay[4]

[1] conforma/cli#3148
[2] konflux-ci/konflux-ci#5689
[3] konflux-ci/tekton-integration-catalog#272
[4] konflux-ci/e2e-tests#1798

Assisted-by: Cursor
Signed-off-by: Lubomir Gallovic <lgallovi@redhat.com>
@querti
Copy link
Author

querti commented Mar 2, 2026

This PR adds the cert to other tasks in the push-to-external-registry pipeline: konflux-ci/release-service-catalog#2031

@querti querti marked this pull request as ready for review March 3, 2026 10:06
@qodo-code-review
Copy link
Contributor

Review Summary by Qodo

Support self-hosted Quay with trusted CA certificates

✨ Enhancement

Grey Divider

Walkthroughs

Description
• Add trusted CA certificate mounting for self-hosted Quay support
• Mount CA bundle to both /mnt/trusted-ca and /etc/ssl/certs/ca-custom-bundle.crt
• Enable Verify Conforma task to work with self-signed certificates
Diagram
flowchart LR
  A["Verify Conforma Task"] -- "mounts CA bundle" --> B["Trusted CA Volume"]
  B -- "maps to /mnt/trusted-ca" --> C["CA Bundle Location 1"]
  B -- "maps to /etc/ssl/certs/ca-custom-bundle.crt" --> D["CA Bundle Location 2"]
  C --> E["Self-hosted Quay Support"]
  D --> E
Loading

Grey Divider

File Changes

1. tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml ✨ Enhancement +7/-0

Add CA certificate mounts for self-hosted Quay

• Added trusted-ca volume mount at /mnt/trusted-ca for consistency with other pipeline tasks
• Added trusted-ca volume mount at /etc/ssl/certs/ca-custom-bundle.crt for system-wide CA trust
• Both mounts are read-only and reference the same ca-bundle.crt subPath
• Enables task to work with self-signed certificates from self-hosted Quay instances

tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml


Grey Divider

Qodo Logo

@qodo-code-review
Copy link
Contributor

qodo-code-review bot commented Mar 3, 2026

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider


Action required

1. CA mount not consumed 🐞 Bug ✓ Correctness
Description
The new stepTemplate mounts the CA bundle into /etc/ssl/certs and /mnt/trusted-ca, but the task does
not set any SSL_CERT_* env for the early use-trusted-artifact step, and the repo’s existing pattern
mounts the bundle under /etc/pki/tls/certs. As a result, the build-trusted-artifacts step may still
fail to trust a self-hosted Quay CA, so the PR may not achieve its stated goal.
Code

tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[R227-233]

+      - name: trusted-ca
+        mountPath: /mnt/trusted-ca
+        readOnly: true
+      - name: trusted-ca
+        mountPath: /etc/ssl/certs/ca-custom-bundle.crt
+        subPath: ca-bundle.crt
+        readOnly: true
Evidence
The PR adds trusted-ca mounts into the stepTemplate at /mnt/trusted-ca and
/etc/ssl/certs/ca-custom-bundle.crt. The step that needs to reach the registry early is
use-trusted-artifact (build-trusted-artifacts image), but the only SSL trust configuration in this
Task is scoped to the later validate step (SSL_CERT_DIR env), and even that step mounts the CA
into /etc/pki/tls/certs (not /etc/ssl/certs). Other tasks in this repo also consistently mount
custom CA under /etc/pki/tls/certs, suggesting /etc/ssl/certs may not be the expected location here.

tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[223-233]
tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[250-257]
tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[348-362]
tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[373-377]
tasks/verify-enterprise-contract/0.1/verify-enterprise-contract.yaml[302-306]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The task now mounts the trusted CA bundle into `/etc/ssl/certs/ca-custom-bundle.crt` and `/mnt/trusted-ca` for all steps, but the step most likely to need it first (`use-trusted-artifact` using the build-trusted-artifacts image) has no explicit SSL trust env configuration. Additionally, the repo’s existing convention mounts custom CA under `/etc/pki/tls/certs/ca-custom-bundle.crt`.

### Issue Context
- `validate` already sets `SSL_CERT_DIR` and mounts CA under `/etc/pki/tls/certs/...`, but earlier steps (notably `use-trusted-artifact`) may still fail against self-hosted Quay.
- The PR’s new mount path `/etc/ssl/certs/...` is inconsistent with other tasks.

### Fix Focus Areas
- tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[223-240]
- tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[250-257]
- tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[348-377]

### Concrete changes to implement
1. Add a stepTemplate env `SSL_CERT_DIR` (or another explicit mechanism your tooling actually honors) so `use-trusted-artifact` can trust the CA. If setting `SSL_CERT_DIR` at stepTemplate level, preserve Tekton’s required defaults as done in `validate` (include `/tekton-custom-certs:/etc/ssl/certs:/etc/pki/tls/certs:/system/etc/security/cacerts` plus any additional paths).
2. Consider also mounting the same CA bundle into `/etc/pki/tls/certs/ca-custom-bundle.crt` in the stepTemplate (matching existing tasks) so both common trust locations are covered consistently across images.
3. Optionally update README to document `/mnt/trusted-ca` usage if it is part of the contract.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. VSA task not updated 🐞 Bug ⛯ Reliability
Description
The PR only updates verify-conforma-konflux-ta; the verify-conforma-konflux-vsa-ta variant still
runs build-trusted-artifacts without mounting trusted-ca in its stepTemplate. If the registry CA
problem affects trusted artifact retrieval (same pattern), pipelines using the VSA task may still
fail against self-hosted Quay.
Code

tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[R227-233]

+      - name: trusted-ca
+        mountPath: /mnt/trusted-ca
+        readOnly: true
+      - name: trusted-ca
+        mountPath: /etc/ssl/certs/ca-custom-bundle.crt
+        subPath: ca-bundle.crt
+        readOnly: true
Evidence
The PR adds trusted-ca mounts to the verify-conforma-konflux-ta stepTemplate, but the VSA variant’s
stepTemplate still mounts only workdir while also using the same build-trusted-artifacts image for
use-trusted-artifact. This indicates the self-hosted Quay CA support is not consistently applied
across closely related tasks in the repo.

tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[223-233]
tasks/verify-conforma-konflux-vsa-ta/0.1/verify-conforma-konflux-vsa-ta.yaml[208-218]
tasks/verify-conforma-konflux-vsa-ta/0.1/verify-conforma-konflux-vsa-ta.yaml[222-229]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Self-hosted Quay support appears to be implemented only for `verify-conforma-konflux-ta`, but the VSA variant `verify-conforma-konflux-vsa-ta` still runs the `use-trusted-artifact` step without mounting the `trusted-ca` ConfigMap (and without equivalent trust configuration).

### Issue Context
Both tasks use the same `quay.io/redhat-appstudio/build-trusted-artifacts:...` image for `use-trusted-artifact`, so if that step needs the custom CA to talk to a self-hosted registry, the VSA variant likely needs the same setup.

### Fix Focus Areas
- tasks/verify-conforma-konflux-vsa-ta/0.1/verify-conforma-konflux-vsa-ta.yaml[208-229]
- tasks/verify-conforma-konflux-ta/0.1/verify-conforma-konflux-ta.yaml[223-233]

### Concrete changes to implement
1. Mirror the `trusted-ca` volumeMounts (and any required SSL trust env settings) from `verify-conforma-konflux-ta` into the `stepTemplate` of `verify-conforma-konflux-vsa-ta`.
2. Ensure the mount paths and env are consistent with the conventions used elsewhere in the repo (notably `/etc/pki/tls/certs/ca-custom-bundle.crt` if that remains the standard).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

ⓘ The new review experience is currently in Beta. Learn more

Grey Divider

Qodo Logo

@querti querti force-pushed the self-hosted-quay branch from c68a985 to ff41ea2 Compare March 3, 2026 10:26
Verify Conforma task should support images that are in a self-hosted
Quay instance. This instance has a self-signed CA cert which is
implicitly not trusted by 3rd party tools. This can be fixed by mounting
the CA bundle to /etc/ssl/certs/, making it trusted. Mount to
/mnt/trusted-ca is added to be consistent with other tasks in the
pipeline.

Assisted-by: Cursor
Signed-off-by: Lubomir Gallovic <lgallovi@redhat.com>
@querti querti force-pushed the self-hosted-quay branch from ff41ea2 to febf62f Compare March 3, 2026 10:34
@querti
Copy link
Author

querti commented Mar 3, 2026

@simonbaird @joejstuart @st3penta can I request a review? Thanks!

- name: trusted-ca
mountPath: /mnt/trusted-ca
readOnly: true
- name: trusted-ca
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intentional to define two different volume mounts with the same name, trusted-ca?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm mounting the same file in two places. Is this a risk or a bad pattern?

Copy link
Member

@simonbaird simonbaird Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh okay I get it. The name refers to the volume, so that's why it's not unique. Lgtm.

@simonbaird
Copy link
Member

In your commit it says "This instance has a self-signed CA cert" which makes me think it was written by Cursor. What instance? Will this change impact Red Hat production Konflux which maybe isn't using a self-signed CA cert?

@querti
Copy link
Author

querti commented Mar 5, 2026

In your commit it says "This instance has a self-signed CA cert" which makes me think it was written by Cursor. What instance? Will this change impact Red Hat production Konflux which maybe isn't using a self-signed CA cert?

Sorry if my original description isn't very clear. We are asked that the release pipeline "push-to-external-registry" should be compatible with self hosted Quay. We will not be using this feature, we will keep using quay.io. This is for our customers who may want to deploy their own quay instance, only available in their intranet. When that is the case, they will have their own CA cert. This CA cert must be mounted in the right place, otherwise tools like skopeo will not trust this instance. This PR mounts the custom CA cert to the right place to ensure that the task works correctly. trusted-ca configmap is generally defined by the cluster admin and should contain the internal CA cert.

I'm not referring to any particular self-hosted quay instance, just any that might exist.

I hope that makes things a bit clearer!

@codecov
Copy link

codecov bot commented Mar 5, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
acceptance 54.84% <ø> (-0.68%) ⬇️
generative 18.16% <ø> (-0.41%) ⬇️
integration 27.00% <ø> (-0.56%) ⬇️
unit 68.65% <ø> (+0.25%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 9 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

querti added a commit to querti/release-service-catalog that referenced this pull request Mar 6, 2026
Add a new e2e test pipeline that checks if push-to-external-registry
works with self-hosted Quay. Unlike other e2e tests which use the
stage cluster, this one uses ephemeral Kind cluster. The reason is
Quay is deployed to the cluster and is used for testing.

Two changes were made to the push-to-external-registry pipeline to
ensure its compatibility with self-hosted Quay:
- Mount the ca-bundle.crt from trusted-ca configmap to
  /etc/ssl/certs/. This ensures that the self-signed CA cert of the
  internal Quay instance is known to the 3rd party tools like skopeo
  and cosign. Without it, the tools would fail with a TLS error.
- Update regexes to support a Quay URL with a port number. The
  regexes assume that only one ":" will be present in the url,
  which may be wrong. Update them to account for hostnames with
  port number.

Several other changes were made to support this functionality:
- Mounting the CA bundle in Conforma task[1]
- Support self-hosted Quay deployment in the Kind cluster[2]
- Add init-quay task and skip-quay parameter[3]
- Add a new Go e2e test for self-hosted quay[4]

[1] conforma/cli#3148
[2] konflux-ci/konflux-ci#5689
[3] konflux-ci/tekton-integration-catalog#272
[4] konflux-ci/e2e-tests#1798

Assisted-by: Cursor
Signed-off-by: Lubomir Gallovic <lgallovi@redhat.com>
@simonbaird
Copy link
Member

One thing I'm wondering about: There's already a mount here just for the validate step but with a different mount path to what you're adding. Is that okay? Will we end up with three mounts available to that step? Do you think it would be possible to remove that mount and let it use the mounts you're added to the template?

Also, along the lines of what Qodo said, do we need to set SSL_CERT_DIR in the stepTemplate also?

@simonbaird
Copy link
Member

One thing I'm wondering about: There's already a mount here just for the validate step but with a different mount path to what you're adding. Is that okay? Will we end up with three mounts available to that step? Do you think it would be possible to remove that mount and let it use the mounts you're added to the template?

I asked Claude about this and it said it should be okay. But do you want to also mount /etc/pki/tls/certs/ca-custom-bundle.crt in the stepTemplate?

Also, along the lines of what Qodo said, do we need to set SSL_CERT_DIR in the stepTemplate also?

I think it could be done later when/if needed, so maybe not a blocker for this PR. Wdyt?

@simonbaird
Copy link
Member

/ok-to-test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants