Running and debugging tests

Running presubmission verification tests

make verify

Running unit tests

To run all unit tests:

make test

To run unit tests for webhooks:

go test ./pkg/webhooks

To run tests that match TestValidateClusterQueue regular expression ref:

go test ./pkg/webhooks -run TestValidateClusterQueue

Running unit tests with race detection

Use -race to enable Go’s built-in race detector:

go test ./pkg/scheduler/preemption/ -race

Running unit tests with stress

To run unit tests in a loop and collect failures:

# install go stress
go install golang.org/x/tools/cmd/stress@latest
# compile tests (you can add -race for race detection)
go test ./pkg/scheduler/preemption/ -c
# it loops and reports failures
stress ./preemption.test -test.run TestPreemption

Running integration tests

make test-integration

For running a subset of tests, see Running subset of tests.

Running e2e tests using custom build

make kind-image-build
make test-e2e
make test-tas-e2e
make test-e2e-customconfigs
make test-e2e-certmanager
make test-e2e-kueueviz
make test-multikueue-e2e

You can also change kind version by modifying E2E_KIND_VERSION variable:

E2E_KIND_VERSION=kindest/node:v1.32.0 make test-e2e

For running a subset of tests, see Running subset of tests.

Increase logging verbosity

You can change log level (for example, set -5 to increase verbosity) using TEST_LOG_LEVEL variables. By default, TEST_LOG_LEVEL=-3.

Debug tests in VSCode

It is possible to debug unit and integration tests in VSCode. You need to have the Go extension installed. Now you will have run test | debug test text buttons above lines like

func TestValidateClusterQueue(t *testing.T) {

You can click on the debug test to debug a specific test.

For integration tests, an additional step is needed. In settings.json, you need to add two variables inside go.testEnvVars:

• Run ENVTEST_K8S_VERSION=1.32 make envtest && ./bin/setup-envtest use $ENVTEST_K8S_VERSION -p path and assign the path to the KUBEBUILDER_ASSETS variable
• Set KUEUE_BIN to the bin directory within your cloned Kueue repository

"go.testEnvVars": {
    "KUBEBUILDER_ASSETS": "<path from output above>",
    "KUEUE_BIN": "<path-to-your-kueue-folder>/bin",
  },

For e2e tests, you can also use Ginkgo Test Explorer. You need to add the following variables to settings.json:

 "ginkgotestexplorer.testEnvVars": {
        "KIND_CLUSTER_NAME": "kind",
        "WORKER1_KIND_CLUSTER_NAME": "kind-worker1",
        "MANAGER_KIND_CLUSTER_NAME": "kind-manager",
        "WORKER2_KIND_CLUSTER_NAME": "kind-worker2",
        "KIND": "<your_kueue_path>/bin/kind",
    },

and then you can use GUI of the Ginkgo Test Explorer to run individual tests, provided you started kind clanter (see here for the instructions).

Attaching e2e tests to an existing kind cluster

You can use the following approach to start up a kind cluster and then run e2e tests from commandline or VSCode, attaching them to the existing cluster. For example, suppose you want to test some of the multikueue-e2e tests. Comment the last line of the hack/multikueue-e2e-test.sh and add read -p "run your tests now" instead:

#$GINKGO $GINKGO_ARGS --junit-report=junit.xml --json-report=e2e.json --output-dir="$ARTIFACTS" -v ./test/e2e/multikueue/...
read -p "run your tests now"

Then, run make kind-image-build test-multikueue-e2e and wait for the "run your tests now" to appear. The cluster is ready, and now you can run tests from another terminal:

<your_kueue_path>/bin/ginkgo --json-report ./ginkgo.report -focus "MultiKueue when Creating a multikueue admission check Should run a jobSet on worker if admitted" -r

or from VSCode.

Running subset of integration or e2e tests

Use Ginkgo –focus arg

GINKGO_ARGS="--focus=Scheduler" make test-integration
GINKGO_ARGS="--focus=Creating a Pod requesting TAS" make test-e2e

Use ginkgo.FIt

If you want to focus on specific tests, you can change ginkgo.It to ginkgo.FIt for these tests. For more details, see here. Then the other tests will be skipped. For example, you can change

ginkgo.It("Should place pods based on the ranks-ordering", func() {

to

ginkgo.FIt("Should place pods based on the ranks-ordering", func() {

and then run

# build and pull image
make test-tas-e2e

to test a particular TAS e2e test.

Use INTEGRATION_TARGET

INTEGRATION_TARGET='test/integration/singlecluster/scheduler' make test-integration

Flaky integration/e2e tests

You can use –until-it-fails or –repeat=N arguments to Ginkgo to run tests repeatedly, such as:

GINKGO_ARGS="--until-it-fails" make test-integration
GINKGO_ARGS="--repeat=10" make test-e2e

See more here

Adding stress

You can run stress tool to increase CPU load during tests. For example, if you’re on Debian-based Linux:

# install stress:
sudo apt install stress
# run stress alongside tests
/usr/bin/stress --cpu 80

Analyzing logs

Kueue runs as a regular pod on a worker node, and in e2e tests there are 2 replicas running. The Kueue logs are located in kind-worker/pods/kueue-system_kueue-controller-manager*/manager and kind-worker2/pods/kueue-system_kueue-controller-manager*/manager folders.

For each log message you can from which file and line the message is coming from:

2025-02-03T15:51:51.502425029Z stderr F 2025-02-03T15:51:51.502117824Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:341	ClusterQueue update event	{"clusterQueue": {"name":"cluster-queue"}}

Here, it’s core/clusterqueue_controller.go:341.

See also

• Kubernetes testing guide
• Integration Testing in Kubernetes
• End-to-End Testing in Kubernetes
• Flaky Tests in Kubernetes