Skip to content

[Bug] Auto-gen matching non-kubernetes resources like `Job`

Kyverno Version

1.12.4

Kubernetes Version 1.29.x

Kubernetes Platform EKS

Kyverno Rule Type Validate

Description

Auto-gen policies are matching non-kubernetes resources like Job from glue.aws.upbound.io API Group. On standard match policies with Pod as the only resource, each policy showed that the auto-gen policy triggered events on the Glue Job resources. We also encountered that when a precondition does not have a non-existence check, that events would stream every few seconds.

To recreate the issue:

  • Install Crossplane and the Upbound provider-family-aws and provider-aws-glue providers
  • Create a glue job managed resource with any Kyverno policy that has Pod as the only matched resource

Glue Job example

apiVersion: glue.aws.upbound.io/v1beta1
kind: Job
metadata:
  annotations:
    meta.upbound.io/example-id: glue/v1beta1/job
  labels:
    testing.upbound.io/example-name: glue-job-example
  name: example
spec:
  forProvider:
    command:
      - scriptLocation: s3://somebucket/example.py
    region: us-east-1
    roleArnRef:
      name: glue-job-example

Pod only policy example

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privilege-escalation
  annotations:
    policies.kyverno.io/title: Disallow Privilege Escalation
    policies.kyverno.io/category: Pod Security Standards (Restricted)
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/severity: high
    policies.kyverno.io/description: >-
      Privilege escalation, such as via set-user-ID or set-group-ID file mode, should not be allowed.
      This policy ensures the `allowPrivilegeEscalation` field is set to `false`.      
spec:
  validationFailureAction: audit
  background: true
  rules:
    - name: privilege-escalation
      match:
        any:
        - resources:
            kinds:
              - Pod    
      validate:
        message: >-
          Privilege escalation is disallowed. The fields
          spec.containers[*].securityContext.allowPrivilegeEscalation,
          spec.initContainers[*].securityContext.allowPrivilegeEscalation,
          and spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
          must be set to `false`.          
        pattern:
          spec:
            =(ephemeralContainers):
            - securityContext:
                allowPrivilegeEscalation: "false"
            =(initContainers):
            - securityContext:
                allowPrivilegeEscalation: "false"
            containers:
            - securityContext:
                allowPrivilegeEscalation: "false"

Precondition without non-existence check example

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-service-account
  annotations:
    policies.kyverno.io/title: Restrict Service Account Only to Jenkins Pods
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/severity: high
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: disallow-service-account
      match:
        resources:
          kinds:
            - Pod
      preconditions:
        any:
          - key: "{{ request.object.spec.serviceAccountName }}"
            operator: Equals
            value: example-account
      validate:        
        message: "Warning! Example-Acccount not allowed on {{` {{ request.image }} `}}!"
        pattern:
          spec:
            containers:
              - image: "*example*"

Screenshot

Screenshot 2024-11-25 at 3 32 33 PM

Expected Behavior

Kyverno should not evaluate non-kubernetes apigroups with auto-gen policies.

Workaround

We were able to workaround this by adding exclude and the non-existence check to the match and precondition statements.

exclude:
  any:
  - resources:
      kinds:
      - glue.aws.upbound.io/*/Job

And added non-existence check to precondition {{ request.object.spec.serviceAccountName || '' }}

Slack discussion

No response

Troubleshooting

  • I have read and followed the documentation AND the troubleshooting guide.
  • I have searched other issues in this repository and mine is not recorded.