Currently, the collator benchmarks separately run through all the strengths, but this makes the suite run for a long time without giving us information beyond the primary level, because the workloads have content that gets decided on the primary level.
We should
- Change the current workloads to run only with a single strength (it shoudn't matter in practice which strength could be primary to emphasize that the workloads are only exercising that strength anyway or tertiary to use the default configuration).
- Introduce new tests that are realistic scenarios that end up with at least some comparisons being decided by the higher strengths being benchmarked.
Currently, the collator benchmarks separately run through all the strengths, but this makes the suite run for a long time without giving us information beyond the primary level, because the workloads have content that gets decided on the primary level.
We should