What is the problem the feature request solves?

I have been experimenting with "immediate mode" shuffle, where we immediately repartition batches using interleave record batches to produce small batches per output partition and then use a BatchCoalescer to combine these small batches into larger batches before encoding.

I am creating this issue to capture lessons learned.

Describe the potential solution

No response

Additional context

No response