Gateway API Service targetPort Not Synchronized with Envoy Port (9999)

Description

Cilium Operator creates a Service for Gateway API with targetPort=80 (from Gateway spec), but Envoy listens on port 9999 (standard port for Cilium Gateway API). Cilium Operator does not synchronize targetPort with the actual Envoy port, causing LoadBalancer health check failures.

Versions

• Cilium: 1.18.3 (quay.io/cilium/cilium:v1.18.3@sha256:5649db451c88d928ea585514746d50d91e6210801b300c897283ea319d68de15)
• Cilium Operator: 1.18.3 (quay.io/cilium/operator-generic:v1.18.3@sha256:b5a0138e1a38e4437c5215257ff4e35373619501f4877dbaf92c89ecfad81797)
• Kubernetes: v1.33.5
• Cloud Provider: Hetzner Cloud
• kube-proxy-replacement: true (Cilium uses eBPF for routing)

Steps to Reproduce

1. Create a Gateway with port 80:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: default
spec:
  gatewayClassName: cilium
  listeners:
  - name: web-gw
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: Same

2. Check the created Service:

kubectl get svc cilium-gateway-my-gateway -n default -o jsonpath='{.spec.ports[0].targetPort}'
# Result: 80 (incorrect)

3. Check the endpoint port:

kubectl get endpoints cilium-gateway-my-gateway -n default -o jsonpath='{.subsets[0].ports[0].port}'
# Result: 9999 (correct - Envoy listens on 9999)

4. Try to fix targetPort manually:

kubectl patch svc cilium-gateway-my-gateway -n default --type='json' \
  -p='[{"op": "replace", "path": "/spec/ports/0/targetPort", "value": 9999}]'

5. Check targetPort again:

kubectl get svc cilium-gateway-my-gateway -n default -o jsonpath='{.spec.ports[0].targetPort}'
# Result: 80 (Cilium Operator overwrote it back)

Expected Behavior

Cilium Operator should automatically synchronize the Service targetPort with the actual Envoy port (9999), so that:

• LoadBalancer health checks work correctly
• Traffic reaches Envoy properly

Actual Behavior

Cilium Operator creates a Service with targetPort=80 (from Gateway spec) and aggressively overwrites any manual changes back to 80, even though Envoy listens on port 9999.

Impact

1. LoadBalancer health check fails (Hetzner Cloud):

   • Hetzner Cloud LoadBalancer performs TCP health check on port 80 (Service port)
   • Health check goes through NodePort → Service port 80 → targetPort=80
   • But Envoy listens on port 9999, not 80
   • Health check cannot reach Envoy → unhealthy status in Hetzner Cloud Console
   • LoadBalancer stops routing traffic to unhealthy targets

2. Traffic may not reach Envoy:

   • Depending on configuration (eBPF may route automatically, but not guaranteed)
   • With kube-proxy-replacement: true, Cilium uses eBPF which might route traffic automatically
   • However, health check still fails because it goes to targetPort=80 instead of 9999

Configuration

• kube-proxy-replacement: true (Cilium uses eBPF)
• gatewayAPI.enabled: true
• Cloud Provider: Hetzner Cloud
  • Hetzner Cloud Controller Manager: Running (2 pods)
  • LoadBalancer type: lb11 (Hetzner Cloud LoadBalancer)
  • LoadBalancer location: nbg1 (Nuremberg)
  • Health check protocol: TCP
  • Health check port: 80 (Service port)
  • Health check annotations: load-balancer.hetzner.cloud/health-check-port: "9999" (configured but not effective)
  • Problem: Health check goes to targetPort (80) instead of the configured health check port (9999) or the actual Envoy port (9999)

Note: This issue may also occur with other cloud providers that use LoadBalancer health checks.

Additional Information

• Manual patching does not work - Cilium Operator overwrites immediately

Hetzner Cloud Specific Details

• LoadBalancer Health Check Behavior:
  • Hetzner Cloud LoadBalancer performs TCP health checks on the Service port (80)
  • Health check flow: LoadBalancer → NodePort (31416) → Service port (80) → targetPort (80)
  • Since targetPort=80 but Envoy listens on 9999, health check fails
  • Health check status in Hetzner Cloud Console: unhealthy
  • LoadBalancer stops routing traffic to unhealthy targets

Workaround

Currently, there is no reliable workaround. The only option is to wait for a fix in a future Cilium version or use a script that continuously patches the Service (not recommended).

Related

• This appears to be a bug in Cilium 1.18.3 Gateway API implementation
• The issue affects LoadBalancer health checks and potentially traffic routing

Recommendation

Cilium Operator should be fixed to synchronize targetPort with the actual Envoy port (9999) instead of using the port from Gateway spec (80).