[SPARK-56415][INFRA] Refactor create_spark_jira.py for LLM use and extract shared utilities

[cloud-fan]

What changes were proposed in this pull request?

Refactors dev/create_spark_jira.py into a lightweight LLM-friendly script, extracts shared JIRA utilities into dev/spark_jira_utils.py, and preserves the original interactive script as dev/create_jira_and_branch.py.

Changes:

• dev/spark_jira_utils.py (new): shared module containing get_jira_client(), detect_affected_version(), list_components(), and create_jira_issue().
• dev/create_spark_jira.py (simplified): stripped down to only create a JIRA ticket and print the key. Made -c (component) required, added --list-components flag, added --parent/--type mutual exclusivity validation, and improved error messages when jira library or JIRA_ACCESS_TOKEN is missing.
• dev/create_jira_and_branch.py (new file, old behavior): the original script that creates a JIRA ticket, checks out a branch, and creates an initial commit — now importing shared utilities from spark_jira_utils.py.
• CLAUDE.md: updated instructions for LLM agents to use the simplified script.

Why are the changes needed?

Previously, CLAUDE.md told LLM agents to ask the user to create JIRA tickets manually. The create_spark_jira.py script existed but included interactive prompts and git side effects (branch creation, committing) that made it unsuitable for automated use. This change makes the script LLM-friendly while preserving the original interactive workflow in a separate script, with shared JIRA logic extracted to avoid duplication.

Does this PR introduce any user-facing change?

No. The original interactive workflow is preserved in dev/create_jira_and_branch.py.

How was this patch tested?

#55280 was created with this new prompt.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

[cloud-fan]

I can add back the old script if people still need it, but with a different name, as create_spark_jira.py should only create ticket.

[cloud-fan]

cc @dongjoon-hyun @HyukjinKwon

[dongjoon-hyun]

Thank you for pinging me. Let me try today.

[dongjoon-hyun]

I'm alway using this parent JIRA ID feature. Please recover this, @cloud-fan .

[cloud-fan]

fixed

[dongjoon-hyun]

I've been considering this improvement. It would be great to preserve all existing features because it's a fallback for non-LLM-installed environment or users, @cloud-fan .

I can add back the old script if people still need it, but with a different name, as create_spark_jira.py should only create ticket.

In addition, it would be great if we can keep the existing file name because create_spark_jira.py is used conventionally by multiple committers across multiple Spark sub-projects already.

• https://github.com/apache/spark/blob/master/dev/create_spark_jira.py
• https://github.com/apache/spark-kubernetes-operator/blob/main/dev/create_spark_jira.py
• https://github.com/apache/spark-connect-swift/blob/main/dev/create_spark_jira.py

It's the same for merge_spark_pr.py:

• https://github.com/apache/spark/blob/master/dev/merge_spark_pr.py
• https://github.com/apache/spark-kubernetes-operator/blob/main/dev/merge_spark_pr.py
• https://github.com/apache/spark-connect-swift/blob/main/dev/merge_spark_pr.py

This PR had better narrow down toward teaching LLMs to skip some existing human-oriented features which is not required for LLM-environment.

[cloud-fan]

I'm fine to create a new script for this llm use case dedicatedly. But I'm confused about the naming suggestion. This simply creates Spark JIRA ticket but we can't name it create_spark_jira.py? It's used by multiple Spark sub-projects does not mean it's correct.

[cloud-fan]

@dongjoon-hyun the old script is kept with a more accurate name, as it does more than jira creation. dev/spark_jira_utils.py is created to share code between the two scripts.

[cloud-fan]

PR description also updated.