Background

Currently, when developing in IntelliJ IDEA, users need to manually select both a Spark profile (e.g., spark-3.5) and a Scala profile (e.g., scala-2.12) to ensure correct compilation. This two-step configuration is error-prone and can lead to compatibility issues if mismatched.

Proposed Change

Automatically lock the Scala version and compiler configuration when a Spark profile is selected, eliminating the need to manually select a separate Scala profile.

Mapping

Implementation Details

For each Spark profile, add:

1. Properties:

   <scalaVersion>2.12</scalaVersion>  <!-- or 2.13 for Spark 4.x -->
   <scalaLongVersion>2.12.18</scalaLongVersion>  <!-- or 2.13.17 for Spark 4.x -->

2. scala-maven-plugin configuration:

• Spark 3.x: Add semanticdb-scalac + paradise compiler plugins
• Spark 4.x: Add -Ymacro-annotations args + semanticdb-scalac plugin (no paradise needed)
• Use combine.self="override" to ensure profile configurations don't merge with base configurations

Benefits

• Simplified workflow: Only select one profile (e.g., spark-3.5) instead of two
• Reduced errors: Eliminates Scala/Spark version mismatch issues
• Better DX: Clearer intent and easier onboarding for new developers

Testing

Verified in IntelliJ IDEA by:

1. Selecting only spark-3.5 profile → Scala 2.12.18 correctly configured
2. Selecting only spark-4.0 profile → Scala 2.13.17 correctly configured
3. No need to select separate scala-2.12 or scala-2.13 profiles