Skip to content

Commit fe54d6a

Browse files
dcalavrezo-qorixdcalavrezo-qorix
authored
coverage: add Ferrocene Rust coverage tool and docs (#102)
* coverage: add Ferrocene Rust coverage tool and docs - add ferrocene_report orchestration and repo-local wrapper rule - support line-coverage parsing and gating - default reports to bazel-bin/coverage and add overall summary - improve path remapping, crate resolutio and runfiles handling - update README/docs for developers and integrators Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com> * update: fix formatiing Fixed formatting, updated version Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com> * changes: comments from Max split the python scripts into separate files added some tests updated worfklow split shell script into a template Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com> * format: fix Fixing formatting issues Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com> --------- Signed-off-by: Dan Calavrezo <195309321+dcalavrezo-qorix@users.noreply.github.com>
1 parent 8894fe5 commit fe54d6a

21 files changed

Lines changed: 1703 additions & 4 deletions

‎.github/workflows/tests.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,6 @@ jobs:
2020
run: |
2121
cd cr_checker/tests
2222
bazel test //...
23+
- name: Run coverage module tests
24+
run: |
25+
bazel test //coverage/tests:all

‎MODULE.bazel‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313

1414
module(
1515
name = "score_tooling",
16-
version = "1.0.5",
16+
version = "1.1.0",
1717
compatibility_level = 1,
1818
)
1919

‎README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ See the individual README files for detailed usage instructions and configuratio
2424
| **python_basics** | Python development utilities and testing | [README](python_basics/README.md) |
2525
| **starpls** | Starlark language server support | [README](starpls/README.md) |
2626
| **tools** | Formatters & Linters | [README](tools/README.md) |
27+
| **coverage** | Ferrocene Rust coverage workflow | [README](coverage/README.md) |
2728

2829
## Usage Examples
2930

@@ -32,6 +33,27 @@ Load tools in your `BUILD` files:
3233
```starlark
3334
load("@score_tooling//:defs.bzl", "score_py_pytest")
3435
load("@score_tooling//:defs.bzl", "cli_tool")
36+
load("@score_tooling//coverage:coverage.bzl", "rust_coverage_report")
37+
```
38+
39+
Create a repo-local coverage target:
40+
41+
```starlark
42+
rust_coverage_report(
43+
name = "rust_coverage",
44+
bazel_configs = [
45+
"ferrocene-x86_64-linux",
46+
"ferrocene-coverage",
47+
],
48+
query = 'kind("rust_test", //...)',
49+
min_line_coverage = "80",
50+
)
51+
```
52+
53+
Then run:
54+
55+
```bash
56+
bazel run //:rust_coverage -- --min-line-coverage 80
3557
```
3658

3759
## Upgrading from separate MODULES
@@ -53,6 +75,7 @@ The available import targets are:
5375
- cli_helper
5476
- use_format_targets
5577
- setup_starpls
78+
- rust_coverage_report
5679

5780
## Format the tooling repository
5881
```bash

‎coverage/BUILD‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# *******************************************************************************
2+
# Copyright (c) 2025 Contributors to the Eclipse Foundation
3+
#
4+
# See the NOTICE file(s) distributed with this work for additional
5+
# information regarding copyright ownership.
6+
#
7+
# This program and the accompanying materials are made available under the
8+
# terms of the Apache License Version 2.0 which is available at
9+
# https://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# SPDX-License-Identifier: Apache-2.0
12+
# *******************************************************************************
13+
14+
load("@rules_shell//shell:sh_binary.bzl", "sh_binary")
15+
16+
package(default_visibility = ["//visibility:public"])
17+
18+
exports_files([
19+
"ferrocene_report_wrapper.sh.tpl",
20+
"scripts/normalize_symbol_report.py",
21+
"scripts/parse_line_coverage.py",
22+
])
23+
24+
sh_binary(
25+
name = "ferrocene_report",
26+
srcs = ["ferrocene_report.sh"],
27+
data = [
28+
"scripts/normalize_symbol_report.py",
29+
"scripts/parse_line_coverage.py",
30+
],
31+
visibility = ["//visibility:public"],
32+
)
33+
34+
sh_binary(
35+
name = "llvm_profile_wrapper",
36+
srcs = ["llvm_profile_wrapper.sh"],
37+
visibility = ["//visibility:public"],
38+
)

‎coverage/README.md‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
# Ferrocene Rust Coverage
2+
3+
This directory provides the Ferrocene Rust coverage workflow for Bazel-based
4+
projects. It uses Ferrocene's `symbol-report` and `blanket` tools to generate
5+
HTML coverage reports from `.profraw` files produced by Rust tests.
6+
7+
The workflow is intentionally split:
8+
- Tests produce `.profraw` files (can run on host or target hardware).
9+
- Reports are generated later on a host machine.
10+
11+
This makes it easy to collect coverage from cross-compiled tests or from
12+
hardware-in-the-loop runs.
13+
14+
## Quick Start (Developers)
15+
16+
1) Run tests with coverage enabled:
17+
18+
```bash
19+
bazel test --config=ferrocene-x86_64-linux --config=ferrocene-coverage \
20+
--nocache_test_results \
21+
//path/to:rust_tests
22+
```
23+
24+
2) Generate coverage reports:
25+
26+
```bash
27+
bazel run //:rust_coverage -- --min-line-coverage 80
28+
```
29+
30+
The default report directory is:
31+
32+
```
33+
$(bazel info bazel-bin)/coverage/rust-tests/<target>/blanket/index.html
34+
```
35+
36+
The script prints per-target line coverage plus an overall summary line.
37+
38+
## Integrator Setup
39+
40+
### 1) MODULE.bazel
41+
42+
Add `score_tooling` and `score_toolchains_rust` as dependencies:
43+
44+
```starlark
45+
bazel_dep(name = "score_tooling", version = "1.0.0")
46+
bazel_dep(name = "score_toolchains_rust", version = "0.4.0")
47+
```
48+
49+
### 2) .bazelrc
50+
51+
Add a Ferrocene coverage config. Names are examples; choose names that fit
52+
your repo:
53+
54+
```
55+
# Ferrocene toolchain for host execution
56+
build:ferrocene-x86_64-linux --host_platform=@score_bazel_platforms//:x86_64-linux
57+
build:ferrocene-x86_64-linux --platforms=@score_bazel_platforms//:x86_64-linux
58+
build:ferrocene-x86_64-linux --extra_toolchains=@score_toolchains_rust//toolchains/ferrocene:ferrocene_x86_64_unknown_linux_gnu
59+
60+
# Coverage flags for rustc
61+
build:ferrocene-coverage --@rules_rust//rust/settings:extra_rustc_flag=-Cinstrument-coverage
62+
build:ferrocene-coverage --@rules_rust//rust/settings:extra_rustc_flag=-Clink-dead-code
63+
build:ferrocene-coverage --@rules_rust//rust/settings:extra_rustc_flag=-Ccodegen-units=1
64+
build:ferrocene-coverage --@rules_rust//rust/settings:extra_rustc_flag=-Cdebuginfo=2
65+
build:ferrocene-coverage --@rules_rust//rust/settings:extra_exec_rustc_flag=-Cinstrument-coverage
66+
build:ferrocene-coverage --@rules_rust//rust/settings:extra_exec_rustc_flag=-Clink-dead-code
67+
build:ferrocene-coverage --@rules_rust//rust/settings:extra_exec_rustc_flag=-Ccodegen-units=1
68+
build:ferrocene-coverage --@rules_rust//rust/settings:extra_exec_rustc_flag=-Cdebuginfo=2
69+
test:ferrocene-coverage --run_under=@score_tooling//coverage:llvm_profile_wrapper
70+
```
71+
72+
### 3) Add a repo-local wrapper target
73+
74+
In a root `BUILD` file:
75+
76+
```starlark
77+
load("@score_tooling//coverage:coverage.bzl", "rust_coverage_report")
78+
79+
rust_coverage_report(
80+
name = "rust_coverage",
81+
bazel_configs = [
82+
"ferrocene-x86_64-linux",
83+
"ferrocene-coverage",
84+
],
85+
query = 'kind("rust_test", //...)',
86+
min_line_coverage = "80",
87+
)
88+
```
89+
90+
Run it with:
91+
92+
```bash
93+
bazel run //:rust_coverage
94+
```
95+
96+
### 4) Optional: exclude known-problematic targets
97+
98+
```starlark
99+
query = 'kind("rust_test", //...) except //path/to:tests',
100+
```
101+
102+
## Cross/Target Execution
103+
104+
If tests run on target hardware, copy the `.profraw` files back to the host
105+
and point the report generator to the directory:
106+
107+
```bash
108+
bazel run //:rust_coverage -- --profraw-dir /path/to/profraw
109+
```
110+
111+
## Coverage Gate Behavior
112+
113+
`--min-line-coverage` applies per target. If any target is below the minimum,
114+
the script exits non-zero so CI can fail the job. An overall summary is printed
115+
for visibility but does not change gating behavior.
116+
117+
## Common Pitfalls
118+
119+
- **"running 0 tests"**: The Rust test harness found no `#[test]` functions,
120+
so coverage is 0%. Add tests or exclude the target from the query.
121+
- **"couldn't find source file"** warnings: Usually path remapping or crate
122+
mapping issues. Check that `crate` attributes in `rust_test` targets point to
123+
the library crate (or exclude the target).
124+
- **Cached test results**: Use `--nocache_test_results` if you need to re-run
125+
tests and regenerate `.profraw` files.
126+
127+
## Troubleshooting
128+
129+
### Coverage is 0% but tests ran
130+
- Verify the target contains real `#[test]` functions. A rust_test target with
131+
no tests will run but report 0% coverage.
132+
- Ensure you ran tests with `--config=ferrocene-coverage` so `.profraw` files
133+
exist.
134+
- If the test binary is cached, use `--nocache_test_results`.
135+
136+
### "couldn't find source file" warnings
137+
- Check `crate` mapping on `rust_test` targets. If `crate = "name"` is used,
138+
ensure it refers to the library crate in the same package.
139+
- Confirm the reported paths exist in the workspace. Path remapping is required
140+
so `blanket` can resolve files under `--ferrocene-src`.
141+
142+
### No `.profraw` files found
143+
- Ensure `test:ferrocene-coverage` sets `--run_under=@score_tooling//coverage:llvm_profile_wrapper`.
144+
- Re-run tests with `--nocache_test_results`.
145+
- If tests ran on target hardware, copy the `.profraw` files back and pass
146+
`--profraw-dir`.
147+
148+
### Coverage gate fails in CI
149+
- The gate is per-target. A single target below the threshold fails the job.
150+
- Use a stricter query (exclude known-zero targets) or add tests.
151+
152+
## CI Integration (Suggested Pattern)
153+
154+
Keep coverage generation separate from docs:
155+
156+
1) Coverage workflow:
157+
- run `bazel run //:rust_coverage`
158+
- upload `bazel-bin/coverage/rust-tests` as an artifact
159+
160+
2) Docs workflow:
161+
- download the artifact
162+
- copy into the docs output (e.g. `docs/_static/coverage/`)
163+
- publish Sphinx docs to GitHub Pages

‎coverage/coverage.bzl‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# *******************************************************************************
2+
# Copyright (c) 2025 Contributors to the Eclipse Foundation
3+
#
4+
# See the NOTICE file(s) distributed with this work for additional
5+
# information regarding copyright ownership.
6+
#
7+
# This program and the accompanying materials are made available under the
8+
# terms of the Apache License Version 2.0 which is available at
9+
# https://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# SPDX-License-Identifier: Apache-2.0
12+
# *******************************************************************************
13+
14+
"""Bazel helpers for Ferrocene Rust coverage workflows."""
15+
16+
def _shell_quote(value):
17+
if value == "":
18+
return "''"
19+
return "'" + value.replace("'", "'\"'\"'") + "'"
20+
21+
def _rust_coverage_report_impl(ctx):
22+
script = ctx.actions.declare_file(ctx.label.name + ".sh")
23+
24+
args = []
25+
for cfg in ctx.attr.bazel_configs:
26+
if cfg:
27+
args.extend(["--bazel-config", cfg])
28+
if ctx.attr.query:
29+
args.extend(["--query", ctx.attr.query])
30+
if ctx.attr.min_line_coverage:
31+
args.extend(["--min-line-coverage", ctx.attr.min_line_coverage])
32+
33+
args_parts = [_shell_quote(a) for a in args]
34+
35+
# The wrapper script forwards preconfigured args and any extra CLI args.
36+
exec_line = "exec \"${ferrocene_report}\""
37+
if args_parts:
38+
exec_line += " \\\n " + " \\\n ".join(args_parts)
39+
exec_line += " \\\n \"$@\""
40+
41+
# Resolve the report script via runfiles for remote/CI compatibility.
42+
runfile_path = ctx.executable._ferrocene_report.short_path
43+
ctx.actions.expand_template(
44+
template = ctx.file._wrapper_template,
45+
output = script,
46+
substitutions = {
47+
"@@RUNFILE@@": _shell_quote(runfile_path),
48+
"@@EXEC_LINE@@": exec_line,
49+
},
50+
is_executable = True,
51+
)
52+
53+
report_runfiles = ctx.attr._ferrocene_report[DefaultInfo].default_runfiles
54+
runfiles = ctx.runfiles(files = [ctx.executable._ferrocene_report]).merge(report_runfiles)
55+
return [DefaultInfo(executable = script, runfiles = runfiles)]
56+
57+
rust_coverage_report = rule(
58+
implementation = _rust_coverage_report_impl,
59+
executable = True,
60+
attrs = {
61+
"bazel_configs": attr.string_list(
62+
default = ["ferrocene-coverage"],
63+
doc = "Bazel configs passed to ferrocene_report.",
64+
),
65+
"query": attr.string(
66+
default = 'kind("rust_test", //...)',
67+
doc = "Bazel query used to discover rust_test targets.",
68+
),
69+
"min_line_coverage": attr.string(
70+
default = "",
71+
doc = "Optional minimum line coverage percentage.",
72+
),
73+
"_ferrocene_report": attr.label(
74+
default = Label("//coverage:ferrocene_report"),
75+
executable = True,
76+
cfg = "exec",
77+
),
78+
"_wrapper_template": attr.label(
79+
default = Label("//coverage:ferrocene_report_wrapper.sh.tpl"),
80+
allow_single_file = True,
81+
),
82+
},
83+
doc = "Creates a repo-local wrapper for Ferrocene Rust coverage reports.",
84+
)

0 commit comments

Comments
 (0)