Skip to content

'auth-tls-verify-client: optional' does not forward with "verify: FAILED" on expired certificate

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.): 1.1.0

Kubernetes version (use kubectl version): version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"f2fbef752f852512e05e85ebba94d05c90ee0354", GitTreeState:"clean", BuildDate:"2021-11-16T17:13:07Z", GoVersion:"go1 .16.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: Azure AKS
  • OS (e.g. from /etc/os-release): AKSUbuntu-1804containerd-2022.01.19
  • How was the ingress-nginx-controller installed:
    • If helm was used then please show output of helm ls -A | grep -i ingress nginx-ingress-nonpci-private ap-system 10 2021-12-03 15:50:08.318981419 +0000 UTC deployed ingress-nginx-4.0.12 1.1.0
    • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
USER-SUPPLIED VALUES USER-SUPPLIED VALUES: controller: admissionWebhooks: enabled: false affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - ingress-nginx - key: app.kubernetes.io/instance operator: In values: - ingress-nginx - key: app.kubernetes.io/component operator: In values: - controller topologyKey: kubernetes.io/hostname weight: 100 config: client-body-timeout: "10" client-header-timeout: "10" enable-access-log-for-default-backend: "true" hide-headers: x-powered-by hsts-max-age: "15768000" keep-alive: "10" large-client-header-buffers: 4 8k proxy-body-size: 100k proxy-headers-hash-bucket-size: "128" proxy-headers-hash-max-size: "512" proxy-real-ip-cidr: *** proxy-send-timeout: "10" proxy-set-headers: ap-system/nginx-ingress-nonpci-private-custom-headers server-tokens: "false" ssl-ciphers: ALL:!EXP:!NULL:!ADH:!LOW:!SSLv2:!SSLv3:!MD5:!RC4 whitelist-source-range: *** extraArgs: default-ssl-certificate: ap-system/nginx-ingress-nonpci-private-tls ingress-class: nginx-nonpci-private ingressClass: nginx-nonpci-private ingressClassResource: controllerValue: k8s.io/nginx-nonpci-private default: false enabled: true name: nginx-nonpci-private replicaCount: 4 service: annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" service.beta.kubernetes.io/azure-load-balancer-internal-subnet: *** enableHttp: false enableHttps: true externalTrafficPolicy: Local loadBalancerIP: *** defaultBackend: enabled: false fullnameOverride: nginx-ingress-nonpci-private
  • Current state of ingress object, if applicable: (see reproduction)

What happened:

When using nginx.ingress.kubernetes.io/auth-tls-verify-client: optional (or optional_no_ca), and the client provides an expired certificate, the Ingress is responding with HTTP 400 and "The SSL certificate error" page.

What you expected to happen:

According to the documentation, it seems like the request should be forwarded to the backend:

When no or an otherwise invalid certificate is provided, the request does not fail, but instead the verification result is sent to the upstream service.

(from https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#client-certificate-authentication )

This seems to be related to a change in OpenSSL that affected nginx. See this nginx ticket for details: https://trac.nginx.org/nginx/ticket/2319#comment:2

The optional_no_ca is expected to allow certificates which are not signed by a known CA, but not certificates which are otherwise invalid as long as nginx knows it. If you want nginx to continue processing of requests with such invalid certificates, consider appropriate ​error_page configuration

How to reproduce it:

  • Prepare CA and an expired certificate
#create ca
openssl genrsa 4096 > ca-key.pem
openssl req -new -x509 -nodes -days 31 -key ca-key.pem -subj "/CN=demo-ca" -out ca.cr

#create a valid client cert
openssl req -newkey rsa:4096 -nodes -days 31 -subj "/CN=demo-client" -addext "extendedKeyUsage=clientAuth" -keyout client-key.pem -out client.csr
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca-key.pem -out client.crt -CAcreateserial

#create an expired client cert
## install faketime for immediate expiry
sudo apt-get install faketime

openssl req -newkey rsa:4096 -nodes -days 31 -subj "/CN=demo-client" -addext "extendedKeyUsage=clientAuth" -keyout client-expired-key.pem -out client-expired.csr
faketime -f '-32d' /bin/bash -c 'openssl x509 -req -in client-expired.csr -CA ca.crt -CAkey ca-key.pem -out client-expired.crt -CAserial ca.srl'
  • Create CA secret kubectl create secret generic demo-ca --from-file=ca.crt=ca.crt
  • Create an ingress with TLS Client Authentication, using the auth-tls-verify-... annotations: kubectl apply -f demo-ingress.yaml demo-ingress.yaml:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: foo-bar
    annotations:
      kubernetes.io/ingress.class: nginx
	  nginx.ingress.kubernetes.io/auth-tls-secret: default/demo-ca
	  nginx.ingress.kubernetes.io/auth-tls-verify-client: optional
  spec:
    ingressClassName: nginx
    rules:
    - host: foo.bar
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: http-svc
              port: 
                number: 80
  • Perform a request with the valid client certificate as a check: curl -H 'Host: foo.bar' localhost --cert client.crt --key client-key.pem -k -> HTTP 200 - OK
  • Perform a request with the expired client certificate: curl -H 'Host: foo.bar' localhost --cert client-expired.crt --key client-expired-key.pem -k ->
400 The SSL certificate error

400 Bad Request

The SSL certificate error
nginx

Now we get the unexpected error page, instead of a response by the backend.

Anything else we need to know:

I'm not sure if this is either a documentation issue, or if the behaviour is intended to be like it was documented (which differs from nginx).