access_log /dev/stdout main annotation causing issue

We are using nginx ingress has load balancer which work both has load balance and request handler We have namespace called ingress-controller which is deployed using helm chart We have around 80+ namespaces which commonly using single IP address has nginx load balancer Each application has its own ingress service which will communicate to external world with main ingress controller.

One of the team members added below annotations to debug his application  annotations:     nginx.ingress.kubernetes.io/configuration-snippet: |       error_log /dev/stderr debug;  # For debug-level logging       access_log /dev/stdout main;   Now due to this annotation whenever pod of any namespace within the cluster got RESTARTED then corrresponding application URL throwing 502 gateway error, because it still pointing OLD POD ENDPOINT instead of new endpoints.

This started failing for all the application, then somehow, we identified and removed given annotation, then all the applications started working as expected.

Question:

1. How to avoid this in future?
2. Why does local ingress service annotation affect globally?
3. What is the root cause of this issue?
4. How to identify any newly added annotation in their local ingress service?

Helm chart we are using ngress-nginx-4.12.1 App version 1.12.1 Configmap data that exists data:   allow-snippet-annotations: "true"   annotations-risk-level: Critical   enable-vts-status: "true"   proxy-body-size: 2000m   use-forwarded-headers: "true"