SSL Passthrough in seperate Ingress only works when removing another

[Lasslos]

What happened:

I have a pod with two services, which look like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/instance: step-certificates
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: step-certificates
    app.kubernetes.io/version: 0.28.4
    helm.sh/chart: step-certificates-1.28.4
  name: step-certificates
  namespace: project-finatix
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 9000
  selector:
    app.kubernetes.io/instance: step-certificates
    app.kubernetes.io/name: step-certificates
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: step-certificates-no-ssl
  namespace: project-finatix
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 9001
  selector:
    app.kubernetes.io/instance: step-certificates
    app.kubernetes.io/name: step-certificates
  type: ClusterIP

These services are exposed through the following ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  labels:
    app.kubernetes.io/instance: step-certificates
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: step-certificates
    app.kubernetes.io/version: 0.28.4
    helm.sh/chart: step-certificates-1.28.4
  name: step-certificates
  namespace: project-finatix
spec:
  ingressClassName: nginx
  rules:
  - host: ca.example.com
    http:
      paths:
      - backend:
          service:
            name: step-certificates
            port:
              number: 443
        path: /
        pathType: ImplementationSpecific
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
  name: step-certificates-ingress-no-ssl
  namespace: project-finatix
spec:
  ingressClassName: nginx
  rules:
  - host: ca.example.com
    http:
      paths:
      - backend:
          service:
            name: step-certificates-no-ssl
            port:
              number: 80
        path: /1.0/crl
        pathType: Exact

When deploying this, both ingresses work as expected except for SSL Passthrough. It is simply not activated.
What fixes it is to remove the non-ssl ingress and immediately add it again. After that, reloading anything has no effect, it just works. This is very unexpected.

Here are some (cleaned) logs that might help.

After deploying the yaml

I1027 10:20:50.551079       7 store.go:443] "Found valid IngressClass" ingress="project-finatix/step-certificates" ingressclass="nginx"
W1027 10:20:50.551878       7 controller.go:1232] Service "project-finatix/step-certificates" does not have any active Endpoint.
I1027 10:20:50.552137       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"project-finatix", Name:"step-certificates", UID:"c0a205a9-fa07-4eb5-933d-ca8942cfcddd", APIVersion:"networking.k8s.io/v1", ResourceVersion:"87097835", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1027 10:20:50.582599       7 store.go:443] "Found valid IngressClass" ingress="project-finatix/step-certificates-ingress-no-ssl" ingressclass="nginx"
I1027 10:20:50.583492       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"project-finatix", Name:"step-certificates-ingress-no-ssl", UID:"8d32e8fd-fd8d-49fb-ad43-2448255c1300", APIVersion:"networking.k8s.io/v1", ResourceVersion:"87097839", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1027 10:20:50.728163       7 controller.go:228] "Backend successfully reloaded"
I1027 10:20:50.728726       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"network", Name:"ingress-nginx-controller-687dd69877-brgt5", UID:"f4e247b4-29bc-46be-b7cc-de07492cb366", APIVersion:"v1", ResourceVersion:"85916824", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1027 10:20:53.750885       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"project-finatix", Name:"step-certificates-ingress-no-ssl", UID:"8d32e8fd-fd8d-49fb-ad43-2448255c1300", APIVersion:"networking.k8s.io/v1", ResourceVersion:"87097894", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1027 10:20:53.770207       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"project-finatix", Name:"step-certificates", UID:"c0a205a9-fa07-4eb5-933d-ca8942cfcddd", APIVersion:"networking.k8s.io/v1", ResourceVersion:"87097895", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W1027 10:20:53.885881       7 controller.go:1232] Service "project-finatix/step-certificates-no-ssl" does not have any active Endpoint.
W1027 10:20:53.885922       7 controller.go:1232] Service "project-finatix/step-certificates" does not have any active Endpoint.
I1027 10:20:53.892033       7 controller.go:214] "Configuration changes detected, backend reload required"
I1027 10:20:54.044149       7 controller.go:228] "Backend successfully reloaded"
I1027 10:20:54.044841       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"network", Name:"ingress-nginx-controller-687dd69877-brgt5", UID:"f4e247b4-29bc-46be-b7cc-de07492cb366", APIVersion:"v1", ResourceVersion:"85916824", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W1027 10:20:57.220644       7 controller.go:1232] Service "project-finatix/step-certificates-no-ssl" does not have any active Endpoint.
W1027 10:20:57.220687       7 controller.go:1232] Service "project-finatix/step-certificates" does not have any active Endpoint.
W1027 10:21:01.097007       7 controller.go:1232] Service "project-finatix/step-certificates-no-ssl" does not have any active Endpoint.

The (https!) request reaches the nginx ingress for some reason

host="ca.finatix.cloud" status=400 x_forward_for="127.0.0.1" bytes_sent=146 http_referrer="-" http_user_agent="curl/8.11.1" req_id="8d7f9e2e3e21d1e8639a8a89da7db644" req_time=0.003 req_method="GET" req_proto="HTTP/2.0" req_path="/" req_query="-" req_length=32 upstream_status=400 upstream_response_time=0.003 upstream_response_length=48 upstream_addr="10.244.2.55:9000" pstream_namespace="project-finatix" upstream_ingress="step-certificates" upstream_service="step-certificates" upstream_port=443

Removing the non-ssl ingress

I1027 10:25:51.342337       7 controller.go:214] "Configuration changes detected, backend reload required"
I1027 10:25:51.529354       7 controller.go:228] "Backend successfully reloaded"
I1027 10:25:51.530549       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"network", Name:"ingress-nginx-controller-687dd69877-brgt5", UID:"f4e247b4-29bc-46be-b7cc-de07492cb366", APIVersion:"v1", ResourceVersion:"85916824", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

Redeploying the yaml

I1027 10:26:16.198012       7 store.go:443] "Found valid IngressClass" ingress="project-finatix/step-certificates-ingress-no-ssl" ingressclass="nginx"
W1027 10:26:16.198964       7 controller.go:663] Ignoring SSL Passthrough for location "/1.0/crl" in server "ca.finatix.cloud"
I1027 10:26:16.199023       7 event.go:377] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"project-finatix", Name:"step-certificates-ingress-no-ssl", UID:"0e88e750-f879-44c1-b3ab-b2ea3a816065", APIVersion:"networking.k8s.io/v1", ResourceVersion:"87099459", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1027 10:26:16.205151       7 controller.go:214] "Configuration changes detected, backend reload required"
I1027 10:26:16.409708       7 controller.go:228] "Backend successfully reloaded"
I1027 10:26:16.410361       7 event.go:377] Event(v1.ObjectReference{Kind:"Pod", Namespace:"network", Name:"ingress-nginx-controller-687dd69877-brgt5", UID:"f4e247b4-29bc-46be-b7cc-de07492cb366", APIVersion:"v1", ResourceVersion:"85916824", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

After these steps, it works. No logs of requests are found after that, which I believe is expected since the requests never reach nginx.

What you expected to happen: SSL Passthrough works out of the box.

NGINX Ingress controller version:

/nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.13.3
  Build:         93851f05e61d99eea49140c9be73499a3cb92ccc
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.27.1

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

kubectl version         
Client Version: v1.33.3
Kustomize Version: v5.6.0
Server Version: v1.33.0

How to reproduce this issue:

Create two ingresses and two services, deploy them both in a yaml.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

• create a cluster using KIND
• install ingress-nginx controller
• deploy a SSL app
• configure both TLS and also PASSTHROUGH for the same app

Not sure what reasons you have for expecting a TLS termination and also a SSL-Passthough concurrently for the same backend. Unless I understood wrong, it makes no sense

[Lasslos]

Hey, thanks for adding steps to reproduce.
Let me correct the last point:

• configure both Plain HTTP and also PASSTHROUGH for the same app.

The reason for this is that the app, a certificate authority, needs mTLS, which requires PASSTHROUGH, as well as HTTP for the address /1.0/crl, where the certificate revocation list is hosted. Most clients actually expect the certificate revocation list to be plain http, since the content itself is signed.
Since there are two different serivces and ingresses, I do not understand why ingress-nginx cares at all about it being the same backend, maybe something stupid went wrong on my end?

[longwuyuan]

I did not ask about plain HTTP. Please read again.
I asked about BOTH.

• TLS Termination and also
• PASSTHROUGH
  concurrently for same backend.

If I understand wrong, then please correct me.

I think you should try

• remove the unrelated annotations of ssl from the second ingress resource that is for port 80
• try with path as / and pathType NOT as ImplementationSpecific

update the results of tests after changes.
do the tests in a kind cluster

[Lasslos]

Sorry, I don't quite understand.
I do not want TLS Termination and PASSTHROUGH, this is not where I encountered the bug?

[longwuyuan]

I do not want TLS Termination and PASSTHROUGH, this is not where I encountered the bug?

ok. so next step is try the suggestions I made and update results.