[Integration tests framework] `TEST_RUN_UNTIL_FAILURE=true` does not stop running tests on failure

[belimawr]

When running integration tests with TEST_RUN_UNTIL_FAILURE=true the integration test framework won't stop running once a failure happens, it just keeps re-running tests, which causes two problems:

• Important test artifacts left by the failed test for debugging are removed from the VM.
• We might miss that a test has failed if we're not watching the tests run closely

The specific command I was using:

TEST_RUN_UNTIL_FAILURE=true AGENT_VERSION="9.4.0-SNAPSHOT" TEST_PLATFORMS="windows/amd64" mage -v integration:single TestFilebeatReceiverLogAsFilestream

Some logs where we can see the failed test, 0 successful tests and another test run starting:

>>> (windows-amd64-2022-default) Test output (non-sudo) (stderr): Running target: Integration:TestOnRemote
>>> (windows-amd64-2022-default) Test output (non-sudo) (stderr): exec: go "list" "-m"
>>> (windows-amd64-2022-default) Test output (non-sudo) (stderr): Found Elastic Beats dir at C:\Users\windows\agent
>>> (windows-amd64-2022-default) Test output (non-sudo) (stderr): Running dependency: main.Build.TestFakeComponent
>>> (windows-amd64-2022-default) Test output (non-sudo) (stderr): exec: go "build" "-v" "-o" "C:\\Users\\windows\\agent\\pkg\\component\\fake\\component\\component.exe" "C:\\Users\\windows\\agent\\pkg\\component\\fake\\component"
>>> (windows-amd64-2022-default) Test output (non-sudo) (stderr): Running dependency: github.com/elastic/elastic-agent/dev-tools/mage.InstallGoTestTools
>>> (windows-amd64-2022-default) Test output (non-sudo) (stderr): exec: go "install" "gotest.tools/gotestsum"
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): >> go test: remote-windows-amd64-2022-default.ess Testing
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): >> ARGS: remote-windows-amd64-2022-default.ess Command: gotestsum --no-color --junitfile-hide-skipped-tests -f standard-verbose --junitfile build/TEST-go-remote-windows-amd64-2022-default.ess.xml --jsonfile build/TEST-go-remote-windows-amd64-2022-default.ess.out.json -- -tags integration -test.shuffle on -test.timeout 2h0m0s -test.run ^(TestFilebeatReceiverLogAsFilestream)$ github.com/elastic/elastic-agent/testing/integration/ess
>>> (windows-amd64-2022-default) Test output (non-sudo) (stderr): exec: gotestsum --no-color --junitfile-hide-skipped-tests -f standard-verbose --junitfile build/TEST-go-remote-windows-amd64-2022-default.ess.xml --jsonfile build/TEST-go-remote-windows-amd64-2022-default.ess.out.json -- -tags integration -test.shuffle on -test.timeout 2h0m0s -test.run ^(TestFilebeatReceiverLogAsFilestream)$ github.com/elastic/elastic-agent/testing/integration/ess
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): exec: gotestsum --no-color --junitfile-hide-skipped-tests -f standard-verbose --junitfile build/TEST-go-remote-windows-amd64-2022-default.ess.xml --jsonfile build/TEST-go-remote-windows-amd64-2022-default.ess.out.json -- -tags integration -test.shuffle on -test.timeout 2h0m0s -test.run ^(TestFilebeatReceiverLogAsFilestream)$ github.com/elastic/elastic-agent/testing/integration/ess
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): -test.shuffle 1775682694358783700
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): === RUN   TestFilebeatReceiverLogAsFilestream
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fixture.go:394: Extracting artifact elastic-agent-9.4.0-SNAPSHOT-windows-x86_64.zip to C:\Users\windows\AppData\Local\Temp\TestFilebeatReceiverLogAsFilestream1675914859
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fixture.go:412: Completed extraction of artifact elastic-agent-9.4.0-SNAPSHOT-windows-x86_64.zip to C:\Users\windows\AppData\Local\Temp\TestFilebeatReceiverLogAsFilestream1675914859
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fixture.go:1184: Components were not modified from the fetched artifact
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): otel_log_as_filestream_test.go:194:
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Error Trace:  C:/Users/windows/agent/testing/integration/ess/otel_log_as_filestream_test.go:71
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): C:/Program Files/Go/src/runtime/asm_amd64.s:1693
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Error:        Not equal:
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): expected: 100
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): actual  : 150
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Messages:     expecting 100 events, got 150
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): otel_log_as_filestream_test.go:194:
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Error Trace:  C:/Users/windows/agent/testing/integration/ess/otel_log_as_filestream_test.go:60
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): C:/Users/windows/agent/testing/integration/ess/otel_log_as_filestream_test.go:194
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Error:        Condition never satisfied
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Test:         TestFilebeatReceiverLogAsFilestream
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Messages:     did not find the expected number of events
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fixture.go:1602: Temporary directory "C:\\Users\\windows\\AppData\\Local\\Temp\\TestFilebeatReceiverLogAsFilestream1675914859" preserved for investigation/debugging
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fs.go:101: Temporary directory saved: C:\Users\windows\agent\build\TestFilebeatReceiverLogAsFilestream2504705962
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): --- FAIL: TestFilebeatReceiverLogAsFilestream (109.06s)
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): FAIL
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): FAIL  github.com/elastic/elastic-agent/testing/integration/ess        109.252s
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): === Failed
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): === FAIL: testing/integration/ess TestFilebeatReceiverLogAsFilestream (109.06s)
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fixture.go:394: Extracting artifact elastic-agent-9.4.0-SNAPSHOT-windows-x86_64.zip to C:\Users\windows\AppData\Local\Temp\TestFilebeatReceiverLogAsFilestream1675914859
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fixture.go:412: Completed extraction of artifact elastic-agent-9.4.0-SNAPSHOT-windows-x86_64.zip to C:\Users\windows\AppData\Local\Temp\TestFilebeatReceiverLogAsFilestream1675914859
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fixture.go:1184: Components were not modified from the fetched artifact
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): otel_log_as_filestream_test.go:194:
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Error Trace:  C:/Users/windows/agent/testing/integration/ess/otel_log_as_filestream_test.go:71
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): C:/Program Files/Go/src/runtime/asm_amd64.s:1693
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Error:        Not equal:
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): expected: 100
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): actual  : 150
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Messages:     expecting 100 events, got 150
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): otel_log_as_filestream_test.go:194:
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Error Trace:  C:/Users/windows/agent/testing/integration/ess/otel_log_as_filestream_test.go:60
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): C:/Users/windows/agent/testing/integration/ess/otel_log_as_filestream_test.go:194
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Error:        Condition never satisfied
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Test:         TestFilebeatReceiverLogAsFilestream
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): Messages:     did not find the expected number of events
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fixture.go:1602: Temporary directory "C:\\Users\\windows\\AppData\\Local\\Temp\\TestFilebeatReceiverLogAsFilestream1675914859" preserved for investigation/debugging
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): fs.go:101: Temporary directory saved: C:\Users\windows\agent\build\TestFilebeatReceiverLogAsFilestream2504705962
>>> (windows-amd64-2022-default) Test output (non-sudo) (stdout): DONE 1 tests, 1 failure in 109.255s
>>> (windows-amd64-2022-default) Test output (non-sudo) (stderr): Error: go test returned a non-zero value: exit status 1
>>> (windows-amd64-2022-default) non-sudo tests failed: Process exited with status 1
>>> Testing completed (0 successful)
>>> Console output written here: build/TEST-go-integration.out
>>> Console JSON output written here: build/TEST-go-integration.out.json
>>> JUnit XML written here: build/TEST-go-integration.xml
>>> Diagnostic output (if present) here: build/diagnostics
exec: go "list" "-m"
Found Elastic Beats dir at /home/tiago/devel/elastic-agent
>>> Creating zip archive of repo to send to remote hosts
>>> (windows-amd64-2022-default) Starting SSH; connect with `ssh -i /home/foo/bar/elastic-agent/.integration-cache/id_rsa windows@1.2.3.4`
>>> (windows-amd64-2022-default) Connected over SSH
>>> (windows-amd64-2022-default) Copying repo
>>> (windows-amd64-2022-default) Running make mage and prepareOnRemote

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[github-actions]

tl;dr: I could not read the issue body due integrity restrictions, but based on the title/symptom and code investigation, the strongest root cause is a Windows integration-runner bug where non-sudo test results are collected using batch.SudoTests, which can hide failures and keep TEST_RUN_UNTIL_FAILURE=true looping.

Recommendation

Fix WindowsRunner.Run to pass the non-sudo package list to runTestsOnWindows in the non-sudo path (pkg/testing/windows/windows.go), then add a unit test that fails with the current code and passes with the fix. Also add a regression guard that validates loop termination behavior when results contain a failure.

Findings

1. TEST_RUN_UNTIL_FAILURE wiring is present and enabled correctly:

   • dev-tools/mage/settings.go:1519-1521 sets IntegrationTest.RunUntilFailure = true.
   • dev-tools/mage/settings_test.go:784-817 confirms env parsing.

2. Loop stop behavior depends on detected failures:

   • magefile.go:2979-2996 reruns until failedCount > 0; otherwise keeps looping when hasRunUntilFailure(cfg) is true.
   • pkg/testing/runner/runner.go:821-849 computes failures from merged package results.

3. Strongest candidate defect (selected from 3 independent investigations):

   • pkg/testing/windows/windows.go:187-194 non-sudo path builds script from tests but calls:

     runTestsOnWindows(..., batch.SudoTests)

   • This should use non-sudo package tests (batch.Tests), not batch.SudoTests.
   • Because failedCount is derived from merged collected outputs, collecting the wrong package set can mask actual failures and keep the until-failure loop running.

4. Related behavior in same file:

   • pkg/testing/windows/windows.go:293-299 logs session execution failure but continues to result collection. That design can be OK if result collection is correct, but combined with wrong test slice it increases risk of invisible failures.

5. Related issues/PR scan:

   • I found historical flaky-test PRs using TEST_RUN_UNTIL_FAILURE (for example #3235) but no clearly matching prior fix for this specific Windows collection path.

Verification

I ran focused local checks to validate the control flow and settings path:

go test ./dev-tools/mage -run 'TestLoadSettings/loads integration test settings from env vars' -count=1

Output:

ok   github.com/elastic/elastic-agent/dev-tools/mage  0.014s

rg -n 'runTestsOnWindows\(|if failedCount > 0|if !hasRunUntilFailure|TEST_RUN_UNTIL_FAILURE' pkg/testing/windows/windows.go magefile.go dev-tools/mage/settings.go

Output includes:

• pkg/testing/windows/windows.go:190 (non-sudo call uses batch.SudoTests)
• magefile.go:2984 (if failedCount > 0)
• magefile.go:2990 (if !hasRunUntilFailure(cfg))
• dev-tools/mage/settings.go:1519 (TEST_RUN_UNTIL_FAILURE parsing)

I also attempted broader package tests in this CI container, but they are not reliable here due missing tooling (gotestsum) and OOM-related compile failures (compile: signal: killed), so the strongest evidence is static code path + focused unit check.

Detailed Action Plan

1. Fix non-sudo package mapping

   • File: pkg/testing/windows/windows.go
   • Function: func (WindowsRunner) Run(...)
   • Change non-sudo call at ~L190 from batch.SudoTests to batch.Tests.

2. Add regression test for Windows runner mapping

   • Add unit test in pkg/testing/windows (new _test.go file).
   • Mock ssh.SSHClient + NewSession/GetFileContents to assert non-sudo result collection iterates over batch.Tests, and sudo collection iterates over batch.SudoTests.
   • Ensure test fails on current code and passes after the fix.

3. Add stop-condition regression test in integration runner layer

   • File: magefile.go tests (or nearby testable helper extraction if needed).
   • Validate that when merged results include at least one failure, integRunner exits with failure path even when run-until-failure is enabled.

4. Docs note (optional but useful)

   • File: docs/test-framework-dev-guide.md
   • Add a short note that current until-failure behavior is validated across OS runners and include Windows coverage in regression tests.

Related Items

Type	Link / Path	Relevance
File	pkg/testing/windows/windows.go:187-194	Likely root cause (wrong package slice used for non-sudo result collection)
File	pkg/testing/windows/windows.go:282-311	Result collection logic depends on passed package slice
File	magefile.go:2979-2996	TEST_RUN_UNTIL_FAILURE loop/stop condition
File	pkg/testing/runner/runner.go:821-849	Failure count derived from merged package outputs
File	dev-tools/mage/settings.go:1519-1521	Env var wiring for run-until-failure
Doc	docs/test-framework-dev-guide.md:429-434	Expected behavior statement (“keep running until it reports a failure”)
PR	#3235	Historical usage of TEST_RUN_UNTIL_FAILURE in flaky-test workflow

Note

🔒 Integrity filtering filtered 10 items

Integrity filtering activated and filtered the following items during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:[Integration tests framework] TEST_RUN_UNTIL_FAILURE=true does not stop running tests on failure #13545 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #13545 (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #4686 (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #6968 (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #5436 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #5435 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #3183 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #3196 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• issue:[Flaky Test]: TestFleetAirGappedUpgradeUnprivileged – Agent status is not online #6968 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• issue:[Flaky Test]: Serverless TestLogIngestionFleetManaged – error starting service #4686 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[belimawr]

/ai What do you mean by "integrity restrictions"? What prevented you from reading the issue body?

[github-actions]

By "integrity restrictions", I mean the workflow’s built-in data-trust policy (DIFC). It only lets me read resources whose integrity level meets a minimum threshold.

For this issue, the MCP response was:

Resource 'issue:elastic/elastic-agent#13545' has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

So what prevented me from reading the issue body was not a missing GitHub permission in the repo; it was the runtime policy blocking low-integrity issue content from being exposed to me in this run.

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:[Integration tests framework] TEST_RUN_UNTIL_FAILURE=true does not stop running tests on failure #13545 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: Mention in Issue

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[belimawr]

@strawgate is there a way to find out why this issue has "low integrity"? I looked at the AI execution logs, but I did not find the reason for the low score.