Add `persistent-drainable` option for `affinity-mode` ingress annotation to support draining sticky server sessions

What this PR does / why we need it:

Legacy web applications that have sticky session affinity are likely to store important application information in memory on the web server associated with the session. This stateful nature can increase application performance by reducing the amount of contextual information necessary to be supplied by the browser or loaded from a persistent database to handle a request. This performance comes with a maintenance cost, where the state on the web server needs to be carefully handled so that users are not disrupted during web server maintenance.

In a traditional model, where web server application management is hands on, sys admins can mark web servers in a "maintenance" or "draining" state to coordinate offloading user sessions from the draining web servers to other servers that are available. This may take several additional requests to the draining web server, but eventually the number of sessions on the web server will approach zero. This allows the admin to safely install OS updates, etc, on the server without disrupting users.

When these applications are ported to Kubernetes, there is a challenge. Kubernetes may dynamically mark a pod for deletion. While pods are given a preStop hook to gracefully terminate, if the pods' service is exposed via an ingress-nginx controller, even with affinity-mode: persistent, they will immediately stop receiving additional requests that may be necessary for gracefully migrating user sessions to other pods. This is because ingress-nginx removes Endpoint objects from the set of available upstreams when their Endpoint condition is no longer Ready, or in other words when it is Terminating.

This PR addresses this problem by adding an affinity-mode: persistent-drainable option. I'll paste my updated documentation for the annotation here:

The annotation nginx.ingress.kubernetes.io/affinity-mode defines the stickiness of a session.

• balanced (default)

  Setting this to balanced will redistribute some sessions if a deployment gets scaled up, therefore rebalancing the load on the servers.

• persistent

  Setting this to persistent will not rebalance sessions to new servers, therefore providing greater stickiness. Sticky sessions will continue to be routed to the same server as long as its Endpoint's condition remains Ready. If the Endpoint stops being Ready, such when a server pod receives a deletion timestamp, sessions will be rebalanced to another server.

• persistent-drainable <-- NEW

  Setting this to persistent-drainable behaves like persistent, but sticky sessions will continue to be routed to the same server as long as its Endpoint's condition remains Serving, even after the server pod receives a deletion timestamp. This allows graceful session draining during the preStop lifecycle hook. New sessions will not be directed to these draining servers and will only be routed to a server whose Endpoint is Ready, except potentially when all servers are draining.

This issue has been discussed since 2018 in nginx/kubernetes-ingress#5962, which should provide further motivation for this feature. You can see that many of those involved in the discussion have resorted to using jcmoraisjr/haproxy-ingress which has a drain-support feature.

While haproxy-ingress might be a viable alternative, I would like to use ingress-nginx for my organization. There is a strong community of support, and hopefully as you will see through reviewing the PR, it is already almost all the way there to supporting this drain support feature, which is critical for the success of my organization's web application.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• CVE Report (Scanner found CVE and adding report)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation only

Which issue/s this PR fixes

I didn't find one specifically on this repo, but nginx/kubernetes-ingress#5962 looks like the OP was originally trying to use this ingress controller, which implicitly is an issue on this repo. 😄

How Has This Been Tested?

I ran the automated go, lua, and e2e tests to make sure I didn't break anything, and I added additional go and lua unit tests for the new feature.

To test that this worked end to end, I created a very basic PHP deployment that echo's back the pod's hostname. You can download it from affinity-test.tgz.

You can get it set up with:

make dev-env # start the ingress-nginx test cluster
tar xfvz affinity-test.tgz
kubectl create namespace affinity-test
kubectl apply -k affinity-test

You can then validate that it's running by running curl localhost

curl localhost
# Hostname: php-hostname-echo-747df85784-jwd79

You can tune the affinity-mode in the affinity-test/deployment.yaml file to what you want. I mostly tested between persistent and persistent-drainable.

Below are the general test cases I ran.

1. persistent-drainable sticks sessions while draining

1. Make sure persistent-drainable is applied
2. Navigate to localhost in your browser
3. Confirm the session is sticky by reloading several times and seeing the same hostname echo'ed
4. Copy the pod's hostname (like php-hostname-echo-747df85784-dvpc8)
5. Delete the pod with kubectl delete pod <your-pod> -n affinity-test
6. While the pod is deleting (around 30 sec), continue to refresh your browser and confirm that it's hitting the same pod
7. After the pod is deleted, confirm that refreshing hits a new pod

2. persistent-drainable avoids draining pods for new sessions

1. Make sure persistent-drainable is applied
2. Run curl localhost in your shell
3. Delete the pod with kubectl delete pod <pod name> -n affinity-test
4. Run curl localhost (from a new shell) many times while the pod is deleting. Confirm it's never the one you deleted

Note

It's possible that you can run curl before the ingress controller picks up the update, so the first one or two requests may still get routed to the old pod.

3. persistent avoids draining pods for new sessions

1. Make sure pesrsistent is applied
2. Run the rest of the steps from test case 2 and confirm that you don't route to the draining pod

4. persistent does not stick sessions while draining

1. Make sure persistent is applied
2. Navigate to localhost in your browser
3. Confirm the session is sticky by reloading several times and seeing the same hostname echo'ed
4. Copy the pod's hostname (like php-hostname-echo-747df85784-dvpc8)
5. Delete the pod with kubectl delete pod <your-pod> -n affinity-test
6. While the pod is deleting (around 30 sec), continue to refresh your browser and confirm that you don't hit the same pod

5. persistent-drainable hits the default-backend after deleting a deployment

1. Make sure persistent-drainable is applied
2. Delete the deployment with kubectl delete deployment php-hostname-echo -n affinity-test
3. Once the pods go down, confirm that curl localhost hits the default backend (the http response will tell you)

Note

While the pods are going down, new upstreams do get traffic sent to the Serving upstreams. This seems acceptable. It seems like a tragic state that the application is in when all of the pods are terminating, and it seems somewhat up in the air with what to do with new traffic. Since the pods are still Serving, it seems ok to continue sending new traffic there until inevitably there are no more pods to serve the new requests.

6. persistent hits the default-backend immediately after deleting a deployment

1. Make sure persistent is applied
2. Delete the deployment with kubectl delete deployment php-hostname-echo -n affinity-test
3. While pods are going down, confirm that curl localhost hits the default backend (the http response will tell you)

7. persistent-drainable warns when all upstreams for a service are draining

1. Make sure persistent-drainable is applied

2. Delete the deployment with kubectl delete deployment php-hostname-echo -n affinity-test

3. Check the logs for the ingress-nginx-controller pod

4. Confirm you see an entry resembling the following

   W0605 23:23:29.792168      11 controller.go:1138] All Endpoints for Service "affinity-test/php-hostname-echo" are draining.

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I've read the CONTRIBUTION guide
• I have added unit and/or e2e tests to cover my changes.
• All new and existing tests passed.