Make irq-load-balancing.crio.io configurable per CPU by andreaskaris · Pull Request #9223 · cri-o/cri-o

andreaskaris · 2025-05-26T18:24:13Z

For more details, see https://issues.redhat.com/browse/RFE-7465 | https://issues.redhat.com/browse/TELCOSTRAT-318

What type of PR is this?

/kind feature

What this PR does / why we need it:

This commit introduces new value housekeeping for annotation irq-load-balancing.crio.io.
When housekeeping is set:

The housekeeping CPU set is injected into the container's environment variables as OPENSHIFT_HOUSEKEEPING_CPUS.
IRQ SMP affinity bits are not disabled on the housekeeping CPUs when adding a new container.
The housekeeping CPUs are chosen as the first CPU inside each container plus its thread siblings.

Reason for this change:

Customer requirements: Customers have requested the ability to manage IRQs more precisely on the CPUs allocated to their guaranteed pods, particularly in high-performance and latency-sensitive environments.
Improved resource utilization: As pod density increases, the need for dedicated CPUs for IRQ handling as well as for guaranteed pods without IRQs increases the overall resource footprint of the entire solution. This enhancement helps mitigate the footprint gap compared to competing container orchestration platforms by allowing IRQs on a small subset (e.g., 1–2 CPUs) within a pinned container.
Greater application control: By allowing targeted IRQ management within a container's CPU set, customers gain more control over performance tuning and workload characteristics.

Which issue(s) this PR fixes:

This enhancement introduces the ability to configure the irq-load-balancing.crio.io annotation to use a new "housekeeping" mode within CRI-O. When set to "housekeeping", it preserves IRQs on the first CPU and its thread siblings in side each container.

Special notes for your reviewer:

It is acknowledged that CPUs where IRQs are not disabled will handle IRQs for the entire system. Customers have reviewed this behavior and confirmed it is acceptable. The added flexibility and improved efficiency are seen as a worthwhile trade-off.

Does this PR introduce a user-facing change?

This commit introduces a new `housekeeping` value for the `irq-load-balancing.crio.io` annotation.

When `housekeeping` is set:
* The housekeeping CPU set is injected into the container's environment variables as `OPENSHIFT_HOUSEKEEPING_CPUS`
* IRQ SMP affinity bits are not disabled on the housekeeping CPUs when adding a new container
* The housekeeping CPUs are chosen as the first CPU within each container plus its thread siblings

Smoketest and smoke test environment (single thread per core)

smoketest.sh

#!/bin/bash

set -eux

cleanup() {
    set +e
    set +u
    echo "Running cleanup..."
    kubectl delete pod qos-demo
    kubectl wait --for=delete pod/qos-demo --timeout=180s
    if [ "$crio_pid" != "" ]; then
    	kill $crio_pid
    fi
    sleep 10
    killall crio || true
    sleep 5
    systemctl restart crio
    wait_cluster_ready

    set +x
    echo "======================"
    echo "For logs, see $tmp_log"
    echo "======================"
}
trap cleanup EXIT

# Adjust this to your cluster, in my case, I need 7 pods with READY 1/1 to be up for the cluster
# to be o.k.
wait_cluster_ready() {
    for i in {0..9}; do
         echo "Waiting for cluster ready, iteration $i"
         sleep 15
	 out=$(timeout 10 kubectl get pods -A | grep "1/1" | wc -l | tr -d '\n')
         if [ "${out}" == "7" ]; then
             return
         else
             echo "Cluster not ready: $out"
	 fi
    done
    exit 1
}

generate_pod_yaml() {
    i=$1
    f="$(mktemp)"
    envsubst < ${DIR}/${pods_yaml[$i]} > $f
    echo $f
}

get_events() {
    i=$1
    sleep 2
    for c in {0..9}; do
        if ! kubectl get events | grep -q "${expected_event[$i]}"; then
            echo "Couldn't find expected event, expected: ${expected_event[$i]}"
        else
            kubectl get events | grep "${expected_event[$i]}"
            return
        fi
        sleep 30
    done
    echo "Couldn't find events after $((i + 1)) tries"
    exit 1
}


DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

declare -A annotations
annotations[0]='annotations:
   irq-load-balancing.crio.io: "housekeeping"'

annotations[1]='annotations:
    irq-load-balancing.crio.io: "disable"'

annotations[2]='annotations:
    irq-load-balancing.crio.io: "true"'

annotations[3]='annotations:
    irq-load-balancing.crio.io: "garbage"'

annotations[4]='annotations:
    irq-load-balancing.crio.io: ""'

declare -A expected_smp_affinity
expected_smp_affinity[0]="3d1f"
expected_smp_affinity[1]="3c0f"
expected_smp_affinity[2]="3c0f"
expected_smp_affinity[3]="3fff"
expected_smp_affinity[4]="3fff"

declare -A expected_event
expected_event[0]=""
expected_event[1]=""
expected_event[2]=""
expected_event[3]=""
expected_event[4]=""

declare -A pods_yaml
pods_yaml[0]="pods.yaml.j2"
pods_yaml[1]="pods.yaml.j2"
pods_yaml[2]="pods.yaml.j2"
pods_yaml[3]="pods.yaml.j2"
pods_yaml[4]="pods.yaml.j2"

expected_reset_affinity="3fff"
affinity_file="/proc/irq/default_smp_affinity"
CRIO="/home/akaris/development/cri-o/bin/crio"
tmp_log="$(mktemp)"

systemctl stop crio
sleep 5
killall crio || true
sleep 5
# Start from clean sheet for SMP affinity ...
echo $expected_reset_affinity > $affinity_file
# Start custom crio ...
${CRIO} 2>&1 | tee "${tmp_log}" | grep smp &
crio_pid=$!
echo "CRIO PID: ${crio_pid}"

wait_cluster_ready

mask=$(cat ${affinity_file} | tr -d '\n')
echo "Starting SMP affinity mask: $mask"
set +x
echo ""
echo ""
echo ""
set -x
for j in {0..1}; do
for i in "${!annotations[@]}"; do
        kubectl delete events --all --timeout=10s
	set +x
	echo "======================="
	echo "Running smoke test ${j}|${i}"
	echo "======================="
	set -x
	export annotation="${annotations[$i]}"
        echo -e "$annotation"
	timeout 15 kubectl apply -f $(generate_pod_yaml $i)
	if [ "${expected_event[$i]}" == "" ]; then
	    kubectl wait --for=condition=Ready pod/qos-demo --timeout=120s
	else
	    kubectl wait --for=condition=Ready pod/qos-demo --timeout=120s || true
	fi
	mask=$(cat ${affinity_file} | tr -d '\n')
	expected_mask="${expected_smp_affinity[$i]}"
	echo "Got mask: $mask, expected mask: $expected_mask"
	if [ "${expected_event[$i]}" == "" ]; then
	    set +x
	    echo "=== HOUSEKEEPING ENV VARS IN CONTAINERS: ==="
	    for cname in qos-demo-ctr qos-demo-ctr-2 qos-demo-ctr-3; do
	       echo "Container $cname"
	       kubectl exec -c $cname qos-demo -- env | grep -i house || true
            done
	    set -x
	    if [ "${mask}" != "${expected_mask}" ]; then
	        exit 1
	    fi
	else
	    get_events $i
	fi
	kubectl delete pod qos-demo
	kubectl wait --for=delete pod/qos-demo --timeout=180s
	mask=$(cat ${affinity_file} | tr -d '\n')
	echo "After reset --- Got mask: $mask, expected mask: $expected_reset_affinity"
	if [ "${mask}" != "${expected_reset_affinity}" ]; then
	    echo "WARN - NOT RESET CORRECTLY BUT MAY BE DUE TO RACE"
	    echo $expected_reset_affinity > $affinity_file
	    # exit 1
	fi
	set +x
	echo ""
	echo ""
	echo ""
	echo ""
	set -x
done
done

pods.yaml.j2

apiVersion: v1
kind: Pod
metadata:
  name: qos-demo
  $annotation
spec:
  hostNetwork: true
  runtimeClassName: performance-performance
  containers:
  - name: qos-demo-ctr
    image: quay.io/akaris/nice-test
    command:
    - "/bin/sleep"
    - "infinity"
    resources:
      limits:
        memory: "1Gi"
        cpu: "4"
      requests:
        memory: "1Gi"
        cpu: "4"
  - name: qos-demo-ctr-2
    image: quay.io/akaris/nice-test
    command:
    - "/bin/sleep"
    - "infinity"
    resources:
      limits:
        memory: "1Gi"
        cpu: "2"
      requests:
        memory: "1Gi"
        cpu: "2"
  - name: qos-demo-ctr-3
    image: quay.io/akaris/nice-test
    command:
    - "/bin/sleep"
    - "infinity"
    resources:
      limits:
        memory: "1Gi"
        cpu: "1200m"
      requests:
        memory: "1Gi"
        cpu: "1200m"

The virtual system that I'm running the smoke test on has:

14 vCPUs (14 cores, no siblings)

Kubelet config:

[root@centos9 pod]# grep -i full /var/lib/kubelet/config.yaml -C4
syncFrequency: 0s
volumeStatsAggPeriod: 0s
cpuManagerPolicy: static
cpuManagerPolicyOptions:
  full-pcpus-only: "true"
cpuManagerReconcilePeriod: 5s
reservedSystemCPUs: 0-3

crio config:

# cat /etc/crio/crio.conf.d/99-runtimes.conf 
[crio.runtime]
infra_ctr_cpuset = "0-3"




# The CRI-O will check the allowed_annotations under the runtime handler and apply high-performance hooks when one of
# high-performance annotations presents under it.
# We should provide the runtime_path because we need to inform that we want to re-use runc binary and we
# do not have high-performance binary under the $PATH that will point to it.
[crio.runtime.runtimes.high-performance]
inherit_default_runtime = true
allowed_annotations = ["cpu-load-balancing.crio.io", "cpu-quota.crio.io", "irq-load-balancing.crio.io", "cpu-c-states.crio.io", "cpu-freq-governor.crio.io"]

Smoketest and smoke test environment (2 thread siblings)

smoketest.sh

expected_smp_affinity[0]="3f3f"

Everything else the same, 7 cores, 2 thread siblings (0-1, 2-3, and so on) on my test VM.

openshift-ci · 2025-05-26T18:24:27Z

Hi @andreaskaris. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bitoku · 2025-05-27T09:57:15Z

/ok-to-test

bitoku · 2025-06-03T14:24:04Z

/retest

codecov · 2025-06-03T16:22:20Z

Codecov Report

❌ Patch coverage is 59.09091% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.46%. Comparing base (7a2820a) to head (ab0176b).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9223      +/-   ##
==========================================
- Coverage   67.04%   66.46%   -0.59%     
==========================================
  Files         202      202              
  Lines       28159    28238      +79     
==========================================
- Hits        18880    18767     -113     
- Misses       7702     7870     +168     
- Partials     1577     1601      +24

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

MarSik · 2025-06-11T07:55:24Z

Andreas proposes

irq-load-balancing.crio.io: "{: , ...}"

But it might be better to implement it using something like this I think

housekeeping-cpus.crio.io/[container-name]: 0,1
irq-load-balancing.crio.io: housekeeping

cri-o would have to automatically determine the housekeeping cpus' siblings too
and inject an environment variable to the containers with the list translated
to machine specific CPU IDs.

This would allow easier reuse of the cpu list with other features like "oc exec" cpus.

andreaskaris · 2025-06-11T08:34:12Z

I'll change this PR according to Martin's suggestion

bitoku · 2025-10-02T10:43:49Z

+// injectHousekeepingEnv adds the HOUSEKEEPING_CPUS environment variable to the container.
+// This allows the container to be aware of which CPUs are designated for housekeeping tasks.
+func injectHousekeepingEnv(specgen *generate.Generator, housekeeping cpuset.CPUSet) error {
+	if specgen == nil || specgen.Config == nil || specgen.Config.Process == nil {
+		return fmt.Errorf("specgen or one of its required elements is nil, specgen: %v", specgen)
+	}
+
+	specgen.Config.Process.Env = append(specgen.Config.Process.Env,
+		fmt.Sprintf("%s=%s", HousekeepingCPUsEnvVar, housekeeping.String()),
+	)
+
+	return nil
+}


We can replace this function to https://pkg.go.dev/github.com/opencontainers/runtime-tools/generate#Generator.AddProcessEnv

When I add:

specgen.AddProcessEnv(HousekeepingCPUsEnvVar, housekeeping.String())

That yields:

$ go test -v . -ginkgo.v -ginkgo.focus='.*setIRQLoadBalancing with housekeeping CPUs.*' (...) [PANICKED] Test Panicked In [It] at: /usr/local/go/src/internal/runtime/maps/runtime_faststr_swiss.go:265 @ 10/03/25 14:04:38.558 assignment to entry in nil map Full Stack Trace github.com/opencontainers/runtime-tools/generate.(*Generator).addEnv(...) /home/akaris/development/cri-o/vendor/github.com/opencontainers/runtime-tools/generate/generate.go:543 github.com/opencontainers/runtime-tools/generate.(*Generator).AddProcessEnv(0xc0000117a0, {0x2b88dbd, 0x11}, {0xc0003c6014, 0x3}) /home/akaris/development/cri-o/vendor/github.com/opencontainers/runtime-tools/generate/generate.go:519 +0x2f2 github.com/cri-o/cri-o/internal/runtimehandlerhooks.injectHousekeepingEnv(0xc0000117a0, {0x2f070b8?}) /home/akaris/development/cri-o/internal/runtimehandlerhooks/high_performance_hooks_linux.go:1576 +0x1a9 github.com/cri-o/cri-o/internal/runtimehandlerhooks.init.func1.6.4(0xc0002edd40, 0xc0001d0e00, 0x0, {0x2b88d35, 0x11}, {0x2b88d46, 0x11}, {0x2b6d115, 0x3}, 0x0) /home/akaris/development/cri-o/internal/runtimehandlerhooks/high_performance_hooks_test.go:392 +0x5ed github.com/cri-o/cri-o/internal/runtimehandlerhooks.init.func1.6.6.5() /home/akaris/development/cri-o/internal/runtimehandlerhooks/high_performance_hooks_test.go:448 +0xe5

Because g.envMap is nil:

func (g *Generator) addEnv(env, key string) { if idx, ok := g.envMap[key]; ok { // The ENV exists in the cache, so change its value in g.Config.Process.Env g.Config.Process.Env[idx] = env } else { // else the env doesn't exist, so add it and add it's index to g.envMap g.Config.Process.Env = append(g.Config.Process.Env, env) g.envMap[key] = len(g.Config.Process.Env) - 1 } }

In the same file, we already have this function here which I based injectHouseKeepingEnv on:

func injectCpusetEnv(specgen *generate.Generator, isolated, shared *cpuset.CPUSet) { spec := specgen.Config spec.Process.Env = append(spec.Process.Env, fmt.Sprintf("%s=%s", IsolatedCPUsEnvVar, isolated.String()), fmt.Sprintf("%s=%s", SharedCPUsEnvVar, shared.String())) }

Given the failure in CI (may be a false positive and I might have to somehow initialize the envmap, but dunno), I'd prefer to not change it

Other fields are normally initialized, so I expected Env also to be initialized. It sounds like a bug in runtime-tools, and definitely not a blocker of this PR.

After a second look ...

In theory, in order to make this work, I have to simply change the initialization in the unit test:

specgen := generate.NewFromSpec( &specs.Spec{ Process: &specs.Process{}, })

However, my problem with this is that there are locations in the code where cri-o calls generate.Generator{} directly, instead of using the New... functions:

[akaris@workstation cri-o (numeric-irq-load-balancing)]$ grep generate.Generator{ -RI internal/lib/restore_test.go: g := generate.Generator{Config: &spec} internal/oci/oci_linux.go: g := &generate.Generator{ internal/runtimehandlerhooks/high_performance_hooks_test.go: return &generate.Generator{ pkg/config/workloads_test.go: g := &generate.Generator{ pkg/config/workloads_test.go: g := &generate.Generator{ [akaris@workstation cri-o (numeric-irq-load-balancing)]$ grep generate.New -RI internal/lib/restore.go: ctrSpec, err := generate.NewFromFile(filepath.Join(ctr.Dir(), "config.json")) internal/lib/checkpoint.go: specgen, err := generate.NewFromFile(configFile) internal/lib/checkpoint.go: g, err := generate.NewFromFile(jsonPath) internal/config/seccomp/seccomp_test.go: generator, err := generate.New("linux") internal/config/seccomp/seccomp_test.go: generator, err := generate.New("linux") internal/config/seccomp/seccomp_test.go: generator, err := generate.New("linux") internal/factory/container/container.go: spec, err := generate.New(runtime.GOOS) internal/runtimehandlerhooks/high_performance_hooks_test.go: specgen := generate.NewFromSpec(

So there's a legit risk that the map may not be initialized.

So this here might work:

$ git diff diff --git a/internal/runtimehandlerhooks/high_performance_hooks_linux.go b/internal/runtimehandlerhooks/high_performance_hooks_linux.go index c1ba7301f6..014758a484 100644 --- a/internal/runtimehandlerhooks/high_performance_hooks_linux.go +++ b/internal/runtimehandlerhooks/high_performance_hooks_linux.go @@ -1569,13 +1569,11 @@ func excludeHousekeepingCPUs(cpus string, housekeepingSiblings cpuset.CPUSet) (c // injectHousekeepingEnv adds the HOUSEKEEPING_CPUS environment variable to the container. // This allows the container to be aware of which CPUs are designated for housekeeping tasks. func injectHousekeepingEnv(specgen *generate.Generator, housekeeping cpuset.CPUSet) error { - if specgen == nil || specgen.Config == nil || specgen.Config.Process == nil { - return fmt.Errorf("specgen or one of its required elements is nil, specgen: %v", specgen) + if specgen == nil { + return fmt.Errorf("specgen is nil, specgen") } - specgen.Config.Process.Env = append(specgen.Config.Process.Env, - fmt.Sprintf("%s=%s", HousekeepingCPUsEnvVar, housekeeping.String()), - ) + specgen.AddProcessEnv(HousekeepingCPUsEnvVar, housekeeping.String()) return nil } diff --git a/internal/runtimehandlerhooks/high_performance_hooks_test.go b/internal/runtimehandlerhooks/high_performance_hooks_test.go index e297e7c43b..d46c32c68a 100644 --- a/internal/runtimehandlerhooks/high_performance_hooks_test.go +++ b/internal/runtimehandlerhooks/high_performance_hooks_test.go @@ -383,11 +383,10 @@ var _ = Describe("high_performance_hooks", func() { Expect(bannedCPUs).To(Equal(expectedBan)) // Also test that injection via injectHousekeepingEnv works correctly. - specgen := generate.Generator{ - Config: &specs.Spec{ + specgen := generate.NewFromSpec( + &specs.Spec{ Process: &specs.Process{}, - }, - } + }) if !housekeepingSiblings.IsEmpty() { err := injectHousekeepingEnv(&specgen, housekeepingSiblings) Expect(err).ToNot(HaveOccurred())

But I feel it's risky. I'll create a helper to initialize the missing spec and stuff manually instead of relying on specgen.AddProcessEnv(

https://pkg.go.dev/github.com/opencontainers/runtime-tools/generate#NewFromSpec
NewFromSpec is deprecated.

my bad, I was looking at the old version. The deprecation was cancelled.
opencontainers/runtime-tools#747

ah yeah https://github.com/opencontainers/runtime-tools/blob/0ea5ed0382a279b30530acccafaf070fefeddafd/generate/generate.go#L267

Precisely because of The recommended replacement does not work because other code paths expect an initialized envCacheMap.. I kind of wished runtime-tools initialized that cache on the fly if it wasn't initialized ...

Anyway, I reported the issue here:

When creating a Generator instance using zero-value initialization (Generator{}), calling AddProcessEnv causes a nil pointer panic. opencontainers/runtime-tools#790
And submitted a fix here:
*Fix nil ptr panic when adding env variables to uninitialized Generator opencontainers/runtime-tools#791

bitoku · 2025-10-03T12:32:15Z

/lgtm cancel

andreaskaris · 2025-10-03T15:40:53Z

@bitoku So, I did the following, now - add this commit:

$ git show HEAD^^
commit 85088529f2dd6178c0659250d4d48fbbae5c3c9f
Author: Andreas Karis <ak.karis@gmail.com>
Date:   Fri Oct 3 17:15:36 2025 +0200

    Fix Generator initialization to properly initialize envMap
    
    The Generator struct contains an internal envMap field that caches
    environment variable positions for performance optimization when adding
    environment variables. This map is critical for the proper operation of
    the AddProcessEnv() and AddMultipleProcessEnv() methods, which rely on
    it to efficiently detect and update duplicate environment variables.
    
    The previous code directly instantiated a Generator struct literal,
    which leaves the envMap field as nil (the zero value for maps in Go).
    When methods like AddProcessEnv() are subsequently called, they attempt
    to access g.envMap[key], which will work on a nil map for reads but
    creates a bug: the code path at line 543 tries to assign to g.envMap,
    which will panic with "assignment to entry in nil map".
    
    While the current code path in createContainerPlatform() doesn't
    immediately call methods that would trigger this panic, the bug
    represents a latent defect that could cause runtime panics if future
    changes call environment variable manipulation methods on this Generator
    instance.
    
    Signed-off-by: Andreas Karis <ak.karis@gmail.com>

diff --git a/internal/oci/oci_linux.go b/internal/oci/oci_linux.go
index 2a0ea717f1..4b3c25bde4 100644
--- a/internal/oci/oci_linux.go
+++ b/internal/oci/oci_linux.go
@@ -15,13 +15,13 @@ import (
 const InfraContainerName = "POD"
 
 func (r *runtimeOCI) createContainerPlatform(c *Container, cgroupParent string, pid int) error {
-       g := &generate.Generator{
-               Config: &rspec.Spec{
+       g := generate.NewFromSpec(
+               &rspec.Spec{
                        Linux: &rspec.Linux{
                                Resources: &rspec.LinuxResources{},
                        },
                },
-       }
+       )
 
        // First, set the cpuset as the one for the infra container.
        // This should be overridden if specified in a workload.
@@ -34,7 +34,7 @@ func (r *runtimeOCI) createContainerPlatform(c *Container, cgroupParent string,
        }
 
        // Mutate our newly created spec to find the customizations that are needed for conmon
-       if err := r.config.Workloads.MutateSpecGivenAnnotations(InfraContainerName, g, c.Annotations()); err != nil {
+       if err := r.config.Workloads.MutateSpecGivenAnnotations(InfraContainerName, &g, c.Annotations()); err != nil {
                return err
        }

With that, I think that all the production code is actually using the generator.New...() methods; and it's just the _test stuff that doesn't:

$ grep generate.Generator{ -RI
internal/lib/restore_test.go:   g := generate.Generator{Config: &spec}
internal/runtimehandlerhooks/high_performance_hooks_test.go:                    return &generate.Generator{
pkg/config/workloads_test.go:                           g := &generate.Generator{
pkg/config/workloads_test.go:                           g := &generate.Generator{

I added another commit for the injectCPUsetEnv:

$ git show
commit 80311951a05bb023a93e2b46c3605b3bb95ecee3 (HEAD -> numeric-irq-load-balancing, andreaskaris/numeric-irq-load-balancing)
Author: Andreas Karis <ak.karis@gmail.com>
Date:   Fri Oct 3 17:24:08 2025 +0200

    HighPerformanceHooks: use specgen.AddProcessEnv in injectCPUsetEnv
    
    Signed-off-by: Andreas Karis <ak.karis@gmail.com>

diff --git a/internal/runtimehandlerhooks/high_performance_hooks_linux.go b/internal/runtimehandlerhooks/high_performance_hooks_linux.go
index dcb54c28e6..aac8f34020 100644
--- a/internal/runtimehandlerhooks/high_performance_hooks_linux.go
+++ b/internal/runtimehandlerhooks/high_performance_hooks_linux.go
@@ -1515,10 +1515,8 @@ func getPodQuotaV2(mng cgroups.Manager) (string, error) {
 }
 
 func injectCpusetEnv(specgen *generate.Generator, isolated, shared *cpuset.CPUSet) {
-       spec := specgen.Config
-       spec.Process.Env = append(spec.Process.Env,
-               fmt.Sprintf("%s=%s", IsolatedCPUsEnvVar, isolated.String()),
-               fmt.Sprintf("%s=%s", SharedCPUsEnvVar, shared.String()))
+       specgen.AddProcessEnv(IsolatedCPUsEnvVar, isolated.String())
+       specgen.AddProcessEnv(SharedCPUsEnvVar, shared.String())
 }

And in the actual commit, I did:

+// injectHousekeepingEnv adds the HOUSEKEEPING_CPUS environment variable to the container.
+// This allows the container to be aware of which CPUs are designated for housekeeping tasks.
+func injectHousekeepingEnv(specgen *generate.Generator, housekeeping cpuset.CPUSet) error {
+       if specgen == nil {
+               return errors.New("specgen is nil, specgen")
+       }
+
+       specgen.AddProcessEnv(HousekeepingCPUsEnvVar, housekeeping.String())
+
+       return nil
+}

And in the unit test:

			specgen := generate.NewFromSpec(&specs.Spec{Process: &specs.Process{}})

The Generator struct contains an internal envMap field that caches environment variable positions for performance optimization when adding environment variables. This map is critical for the proper operation of the AddProcessEnv() and AddMultipleProcessEnv() methods, which rely on it to efficiently detect and update duplicate environment variables. The previous code directly instantiated a Generator struct literal, which leaves the envMap field as nil (the zero value for maps in Go). When methods like AddProcessEnv() are subsequently called, they attempt to access g.envMap[key], which will work on a nil map for reads but creates a bug: the code path at line 543 tries to assign to g.envMap, which will panic with "assignment to entry in nil map". While the current code path in createContainerPlatform() doesn't immediately call methods that would trigger this panic, the bug represents a latent defect that could cause runtime panics if future changes call environment variable manipulation methods on this Generator instance. Signed-off-by: Andreas Karis <ak.karis@gmail.com>

Add support for the "housekeeping" annotation value in the IRQ load balancing feature. When irq-load-balancing.crio.io is set to "housekeeping", IRQ interrupts are preserved on the first CPU and its thread siblings, while being disabled on the remaining container CPUs. The implementation also injects environment variable HOUSEKEEPING_CPUS into containers. Signed-off-by: Andreas Karis <ak.karis@gmail.com>

Add a nil pointer check in isContainerRequestWholeCPU, just for safety measures, as the cSpec or several of the fields might be nil. Signed-off-by: Andreas Karis <ak.karis@gmail.com>

openshift-ci · 2025-10-06T10:07:26Z

@andreaskaris: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/ci-e2e-evented-pleg	`da15120`	link	false	`/test ci-e2e-evented-pleg`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

andreaskaris · 2025-10-07T20:19:44Z

/retest

andreaskaris · 2025-10-07T20:21:39Z

Thanks for the reviews so far. I'm just wondering if I addressed all of your concerns @bitoku @bartwensley , otherwise please lmk and I'm happy to make the required changes

bitoku

/lgtm

I think it's better to get LGTM from @MarSik or @bartwensley .

MarSik

/lgtm

openshift-ci · 2025-10-08T10:50:11Z

@MarSik: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bitoku · 2025-10-08T10:52:04Z

@cri-o/cri-o-maintainers PTAL for approval.

bartwensley · 2025-10-08T11:40:35Z

/lgtm

openshift-ci · 2025-10-08T11:40:38Z

@bartwensley: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

andreaskaris · 2025-10-08T12:12:12Z

/retest

haircommander · 2025-10-08T15:47:44Z

/approve

thank you!

openshift-ci · 2025-10-08T15:47:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreaskaris, haircommander

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haircommander · 2025-11-03T14:07:11Z

/cherry-pick release-1.34

openshift-cherrypick-robot · 2025-11-03T14:08:13Z

@haircommander: new pull request created: #9564

Details

In response to this:

/cherry-pick release-1.34

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

andreaskaris requested a review from mrunalp as a code owner May 26, 2025 18:24

openshift-ci bot requested review from klihub and littlejawa May 26, 2025 18:24

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 26, 2025

openshift-ci bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels May 26, 2025

andreaskaris force-pushed the numeric-irq-load-balancing branch from 2ef2c88 to 6b73368 Compare May 26, 2025 18:52

bitoku reviewed May 27, 2025

View reviewed changes

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 27, 2025

andreaskaris force-pushed the numeric-irq-load-balancing branch 5 times, most recently from 0e59050 to 5d58c45 Compare May 27, 2025 20:27

bitoku reviewed May 28, 2025

View reviewed changes

Comment thread pkg/annotations/annotations.go Outdated

andreaskaris force-pushed the numeric-irq-load-balancing branch 3 times, most recently from 758735e to 674c9a6 Compare May 28, 2025 21:19

andreaskaris force-pushed the numeric-irq-load-balancing branch 2 times, most recently from ce8f83c to 34ad65a Compare June 12, 2025 11:40

bitoku reviewed Oct 2, 2025

View reviewed changes

andreaskaris force-pushed the numeric-irq-load-balancing branch from 20cfba2 to f33ce9f Compare October 3, 2025 12:11

openshift-ci bot assigned bitoku Oct 3, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 3, 2025

bitoku reviewed Oct 3, 2025

View reviewed changes

Comment thread internal/runtimehandlerhooks/high_performance_hooks_linux.go Outdated

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 3, 2025

andreaskaris force-pushed the numeric-irq-load-balancing branch 2 times, most recently from f02ac74 to 8031195 Compare October 3, 2025 15:26

andreaskaris force-pushed the numeric-irq-load-balancing branch from 8031195 to 0da5659 Compare October 3, 2025 15:44

bartwensley reviewed Oct 3, 2025

View reviewed changes

andreaskaris added 3 commits October 6, 2025 10:38

HighPerformanceHooks: Nil pointer check for isContainerRequestWholeCPU

ab0176b

Add a nil pointer check in isContainerRequestWholeCPU, just for safety measures, as the cSpec or several of the fields might be nil. Signed-off-by: Andreas Karis <ak.karis@gmail.com>

bitoku reviewed Oct 8, 2025

View reviewed changes

MarSik reviewed Oct 8, 2025

View reviewed changes

openshift-cherrypick-robot mentioned this pull request Nov 3, 2025

[release-1.34] Make irq-load-balancing.crio.io configurable per CPU #9564

Merged

Conversation

andreaskaris commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Smoketest and smoke test environment (single thread per core)

Smoketest and smoke test environment (2 thread siblings)

Uh oh!

openshift-ci bot commented May 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bitoku commented May 27, 2025

Uh oh!

Uh oh!

bitoku commented Jun 3, 2025

Uh oh!

codecov bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

MarSik commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreaskaris commented Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bitoku Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

andreaskaris Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bitoku Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

andreaskaris Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

bitoku Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

andreaskaris Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

andreaskaris Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bitoku commented Oct 3, 2025

Uh oh!

andreaskaris commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci bot commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreaskaris commented Oct 7, 2025

Uh oh!

andreaskaris commented Oct 7, 2025

Uh oh!

bitoku left a comment

Choose a reason for hiding this comment

andreaskaris commented May 26, 2025 •

edited

Loading

codecov bot commented Jun 3, 2025 •

edited

Loading

MarSik commented Jun 11, 2025 •

edited

Loading

andreaskaris Oct 3, 2025 •

edited

Loading

andreaskaris commented Oct 3, 2025 •

edited

Loading

openshift-ci bot commented Oct 6, 2025 •

edited

Loading