Skip to content

Commit 32ae5ff

Browse files
shreyas-goenkashreyas-goenka
committed
Integrate deployment metadata service for server-side locking and state
Add server-side deployment locking and state management via the Deployment Metadata Service (DMS), gated behind DATABRICKS_BUNDLE_MANAGED_STATE=true. Key changes: - DeploymentLock interface with factory (DMS or filesystem based on env) - DMS lock: version-based locking with heartbeat, operation reporting - State read/write via ListResources/CreateOperation with per-resource state - withDeploymentLock helper extracts lock boilerplate from deploy/destroy - Temporary DMS client (libs/tmpdms) mirroring future SDK-generated code - Mock DMS server for acceptance tests - 6 acceptance tests covering deploy, destroy, plan, summary, sequential deploys, and adding resources with remote state Co-authored-by: Isaac
1 parent fa37f5c commit 32ae5ff

57 files changed

Lines changed: 2826 additions & 259 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎acceptance/bin/python3‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/usr/bin/python3.12

‎acceptance/bundle/dms/add-resources/databricks.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
bundle:
2+
name: add-resources-test
3+
4+
resources:
5+
jobs:
6+
job_a:
7+
name: job-a

‎acceptance/bundle/dms/add-resources/out.requests.txt‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
{
2+
"method": "GET",
3+
"path": "/.well-known/databricks-config"
4+
}
5+
{
6+
"method": "GET",
7+
"path": "/api/2.0/workspace/get-status",
8+
"q": {
9+
"path": "/Workspace/Users/[USERNAME]/.bundle/add-resources-test/default/state/resources.json",
10+
"return_export_info": "true"
11+
}
12+
}
13+
{
14+
"method": "GET",
15+
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/add-resources-test/default/state/resources.json"
16+
}
17+
{
18+
"method": "GET",
19+
"path": "/api/2.0/bundle/deployments/[UUID]/resources",
20+
"q": {
21+
"page_size": "1000"
22+
},
23+
"body": null
24+
}
25+
{
26+
"method": "GET",
27+
"path": "/api/2.0/workspace/get-status",
28+
"q": {
29+
"path": "/Workspace/Users/[USERNAME]/.bundle/add-resources-test/default"
30+
}
31+
}
32+
{
33+
"method": "GET",
34+
"path": "/api/2.0/workspace/get-status",
35+
"q": {
36+
"path": "/Workspace/Users/[USERNAME]/.bundle/add-resources-test/default/state/resources.json",
37+
"return_export_info": "true"
38+
}
39+
}
40+
{
41+
"method": "GET",
42+
"path": "/api/2.0/workspace-files/Workspace/Users/[USERNAME]/.bundle/add-resources-test/default/state/resources.json"
43+
}
44+
{
45+
"method": "POST",
46+
"path": "/api/2.0/bundle/deployments",
47+
"q": {
48+
"deployment_id": "[UUID]"
49+
},
50+
"body": {
51+
"target_name": "default"
52+
}
53+
}
54+
{
55+
"method": "GET",
56+
"path": "/api/2.0/bundle/deployments/[UUID]"
57+
}
58+
{
59+
"method": "POST",
60+
"path": "/api/2.0/bundle/deployments/[UUID]/versions",
61+
"q": {
62+
"version_id": "3"
63+
},
64+
"body": {
65+
"cli_version": "[DEV_VERSION]",
66+
"version_type": "VERSION_TYPE_DESTROY",
67+
"target_name": "default"
68+
}
69+
}
70+
{
71+
"method": "GET",
72+
"path": "/api/2.2/jobs/get",
73+
"q": {
74+
"job_id": "[NUMID]"
75+
}
76+
}
77+
{
78+
"method": "GET",
79+
"path": "/api/2.2/jobs/get",
80+
"q": {
81+
"job_id": "[NUMID]"
82+
}
83+
}
84+
{
85+
"method": "POST",
86+
"path": "/api/2.2/jobs/delete",
87+
"body": {
88+
"job_id": [NUMID]
89+
}
90+
}
91+
{
92+
"method": "POST",
93+
"path": "/api/2.2/jobs/delete",
94+
"body": {
95+
"job_id": [NUMID]
96+
}
97+
}
98+
{
99+
"method": "POST",
100+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/3/operations",
101+
"q": {
102+
"resource_key": "jobs.job_a"
103+
},
104+
"body": {
105+
"resource_key": "jobs.job_a",
106+
"action_type": "OPERATION_ACTION_TYPE_DELETE",
107+
"resource_id": "[NUMID]",
108+
"status": "OPERATION_STATUS_SUCCEEDED"
109+
}
110+
}
111+
{
112+
"method": "POST",
113+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/3/operations",
114+
"q": {
115+
"resource_key": "jobs.job_b"
116+
},
117+
"body": {
118+
"resource_key": "jobs.job_b",
119+
"action_type": "OPERATION_ACTION_TYPE_DELETE",
120+
"resource_id": "[NUMID]",
121+
"status": "OPERATION_STATUS_SUCCEEDED"
122+
}
123+
}
124+
{
125+
"method": "POST",
126+
"path": "/api/2.0/workspace/delete",
127+
"body": {
128+
"path": "/Workspace/Users/[USERNAME]/.bundle/add-resources-test/default",
129+
"recursive": true
130+
}
131+
}
132+
{
133+
"method": "POST",
134+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/3/complete",
135+
"body": {
136+
"name": "deployments/[UUID]/versions/3",
137+
"completion_reason": "VERSION_COMPLETE_SUCCESS"
138+
}
139+
}
140+
{
141+
"method": "DELETE",
142+
"path": "/api/2.0/bundle/deployments/[UUID]"
143+
}

‎acceptance/bundle/dms/add-resources/out.test.toml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/dms/add-resources/output.txt‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
2+
>>> [CLI] bundle deploy
3+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/add-resources-test/default/files...
4+
Deploying resources...
5+
Deployment complete!
6+
7+
>>> [CLI] bundle deploy
8+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/add-resources-test/default/files...
9+
Deploying resources...
10+
Deployment complete!
11+
12+
>>> [CLI] bundle plan
13+
Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
14+
15+
>>> print_requests.py --get //bundle
16+
{
17+
"method": "POST",
18+
"path": "/api/2.0/bundle/deployments",
19+
"q": {
20+
"deployment_id": "[UUID]"
21+
},
22+
"body": {
23+
"target_name": "default"
24+
}
25+
}
26+
{
27+
"method": "GET",
28+
"path": "/api/2.0/bundle/deployments/[UUID]"
29+
}
30+
{
31+
"method": "POST",
32+
"path": "/api/2.0/bundle/deployments/[UUID]/versions",
33+
"q": {
34+
"version_id": "1"
35+
},
36+
"body": {
37+
"cli_version": "[DEV_VERSION]",
38+
"version_type": "VERSION_TYPE_DEPLOY",
39+
"target_name": "default"
40+
}
41+
}
42+
{
43+
"method": "POST",
44+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/1/operations",
45+
"q": {
46+
"resource_key": "jobs.job_a"
47+
},
48+
"body": {
49+
"resource_key": "jobs.job_a",
50+
"action_type": "OPERATION_ACTION_TYPE_CREATE",
51+
"state": {
52+
"deployment": {
53+
"kind": "BUNDLE",
54+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/add-resources-test/default/state/metadata.json"
55+
},
56+
"edit_mode": "UI_LOCKED",
57+
"format": "MULTI_TASK",
58+
"max_concurrent_runs": 1,
59+
"name": "job-a",
60+
"queue": {
61+
"enabled": true
62+
}
63+
},
64+
"resource_id": "[NUMID]",
65+
"status": "OPERATION_STATUS_SUCCEEDED"
66+
}
67+
}
68+
{
69+
"method": "POST",
70+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/1/complete",
71+
"body": {
72+
"name": "deployments/[UUID]/versions/1",
73+
"completion_reason": "VERSION_COMPLETE_SUCCESS"
74+
}
75+
}
76+
{
77+
"method": "GET",
78+
"path": "/api/2.0/bundle/deployments/[UUID]/resources",
79+
"q": {
80+
"page_size": "1000"
81+
},
82+
"body": null
83+
}
84+
{
85+
"method": "POST",
86+
"path": "/api/2.0/bundle/deployments",
87+
"q": {
88+
"deployment_id": "[UUID]"
89+
},
90+
"body": {
91+
"target_name": "default"
92+
}
93+
}
94+
{
95+
"method": "GET",
96+
"path": "/api/2.0/bundle/deployments/[UUID]"
97+
}
98+
{
99+
"method": "POST",
100+
"path": "/api/2.0/bundle/deployments/[UUID]/versions",
101+
"q": {
102+
"version_id": "2"
103+
},
104+
"body": {
105+
"cli_version": "[DEV_VERSION]",
106+
"version_type": "VERSION_TYPE_DEPLOY",
107+
"target_name": "default"
108+
}
109+
}
110+
{
111+
"method": "POST",
112+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/2/operations",
113+
"q": {
114+
"resource_key": "jobs.job_b"
115+
},
116+
"body": {
117+
"resource_key": "jobs.job_b",
118+
"action_type": "OPERATION_ACTION_TYPE_CREATE",
119+
"state": {
120+
"deployment": {
121+
"kind": "BUNDLE",
122+
"metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/add-resources-test/default/state/metadata.json"
123+
},
124+
"edit_mode": "UI_LOCKED",
125+
"format": "MULTI_TASK",
126+
"max_concurrent_runs": 1,
127+
"name": "job-b",
128+
"queue": {
129+
"enabled": true
130+
}
131+
},
132+
"resource_id": "[NUMID]",
133+
"status": "OPERATION_STATUS_SUCCEEDED"
134+
}
135+
}
136+
{
137+
"method": "POST",
138+
"path": "/api/2.0/bundle/deployments/[UUID]/versions/2/complete",
139+
"body": {
140+
"name": "deployments/[UUID]/versions/2",
141+
"completion_reason": "VERSION_COMPLETE_SUCCESS"
142+
}
143+
}
144+
{
145+
"method": "GET",
146+
"path": "/api/2.0/bundle/deployments/[UUID]/resources",
147+
"q": {
148+
"page_size": "1000"
149+
},
150+
"body": null
151+
}

‎acceptance/bundle/dms/add-resources/script‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Deploy with one job.
2+
trace $CLI bundle deploy
3+
4+
# Delete local cache to force reading state from DMS.
5+
rm -rf .databricks
6+
7+
# Add a second job and deploy again.
8+
cat > databricks.yml << 'EOF'
9+
bundle:
10+
name: add-resources-test
11+
12+
resources:
13+
jobs:
14+
job_a:
15+
name: job-a
16+
job_b:
17+
name: job-b
18+
EOF
19+
trace $CLI bundle deploy
20+
21+
# Delete local cache again and run plan — should show no changes.
22+
rm -rf .databricks
23+
trace $CLI bundle plan
24+
25+
# Print metadata service requests. Should show:
26+
# - Deploy 1: CREATE for job_a
27+
# - Deploy 2: ListResources + CREATE for job_b (job_a is unchanged)
28+
# - Plan: ListResources (no operations)
29+
trace print_requests.py --get //bundle
30+
31+
# Clean up.
32+
rm -rf .databricks
33+
$CLI bundle destroy --auto-approve > /dev/null 2>&1

‎acceptance/bundle/dms/add-resources/test.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Ignore = [".databricks"]

‎acceptance/bundle/dms/databricks.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
bundle:
2+
name: metadata-service-test
3+
4+
resources:
5+
jobs:
6+
test_job:
7+
name: test-job

‎acceptance/bundle/dms/deploy-error/databricks.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
bundle:
2+
name: metadata-service-error-test
3+
4+
resources:
5+
jobs:
6+
test_job:
7+
name: test-job

‎acceptance/bundle/dms/deploy-error/out.test.toml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)