Skip to content

Commit f53d114

Browse files
petetomasikpetetomasik
authored
Merge pull request #149 from buildkite-plugins/SUP-4987-add-azure-backend
Support Azure Blob Storage as cache backend
2 parents 20d17cc + 246bd8b commit f53d114

File tree

3 files changed

+674
-0
lines changed

3 files changed

+674
-0
lines changed

‎README.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ The plugin supports various backends and compression algorithms, and some enviro
3737
- `zip/unzip` for zip compression
3838
- `aws` AWS CLI for reading and writing to an AWS S3 backend
3939
- `gsutil` or `gcloud storage` Google Cloud SDK CLI for reading and writing to a GCS backend
40+
- `az` Azure CLI for reading and writing to an Azure Blob Storage backend
4041

4142
## Mandatory parameters
4243

@@ -64,6 +65,7 @@ Defines how the cache is stored and restored. Can be any string (see [Customizab
6465
* `fs` (default)
6566
* `s3`
6667
* `gcs`
68+
* `azure`
6769

6870
#### `fs`
6971

@@ -164,6 +166,128 @@ steps:
164166
compression: zstd
165167
```
166168

169+
#### `azure`
170+
171+
Store cache in an Azure Blob Storage container. You need to make sure the `az` command is available and appropriately configured with the necessary credentials and access permissions.
172+
173+
**Azure Authentication**: The backend supports multiple authentication methods. Azure CLI will automatically detect and use available credentials, or you can explicitly specify the authentication method.
174+
175+
**Methods with explicit auth-mode support:**
176+
- **Microsoft Entra ID (formerly Azure AD)**: Use `az login` and optionally set `BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE=login` to enforce this method
177+
- **Storage Account Key**: Set `AZURE_STORAGE_KEY` environment variable and optionally set `BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE=key` to enforce this method
178+
179+
**Methods using auto-detection only:**
180+
- **Connection String**: Set `AZURE_STORAGE_CONNECTION_STRING` environment variable (highest priority in auto-detection)
181+
- **SAS Token**: Set `AZURE_STORAGE_SAS_TOKEN` environment variable
182+
183+
You also need the agent to have access to the following defined environment variables:
184+
* `BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER`: the container name to use (backend will fail if not defined)
185+
* `BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT`: the storage account name (backend will fail if not defined)
186+
* `BUILDKITE_PLUGIN_AZURE_CACHE_PREFIX`: optional prefix to use for cache keys within the container
187+
* `BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE`: optional - set to `key` or `login` to enforce a specific authentication method (useful when multiple credentials are present)
188+
* `BUILDKITE_PLUGIN_AZURE_CACHE_QUIET`: optional - set to `true` to suppress progress bars and JSON metadata output from Azure CLI operations (error messages are still displayed). Defaults to `false`.
189+
190+
#### Example with Storage Account Key
191+
192+
```yaml
193+
env:
194+
BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER: "my-cache-container" # Required: Azure container to store cache objects
195+
BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT: "mystorageaccount" # Required: Azure storage account name
196+
BUILDKITE_PLUGIN_AZURE_CACHE_PREFIX: "buildkite/cache"
197+
BUILDKITE_PLUGIN_AZURE_CACHE_QUIET: "true"
198+
BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE: "key" # Use storage account key authentication
199+
200+
steps:
201+
- label: ':nodejs: Install dependencies'
202+
command: npm ci
203+
secrets:
204+
- AZURE_STORAGE_KEY
205+
plugins:
206+
- cache#v1.9.0:
207+
backend: azure
208+
path: node_modules
209+
manifest: package-lock.json
210+
restore: file
211+
save: file
212+
compression: zstd
213+
```
214+
215+
#### Example with Microsoft Entra ID
216+
217+
**Note**: Microsoft Entra ID authentication requires authenticating with Azure before using the cache plugin. You can use `az login` in your agent setup, or use a plugin like [azure-login](https://github.com/buildkite-plugins/azure-login-buildkite-plugin) to authenticate with a managed identity. The authenticated identity must have the appropriate RBAC role (such as "Storage Blob Data Contributor") assigned to the storage account.
218+
219+
```yaml
220+
env:
221+
BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER: "my-cache-container" # Required: Azure container to store cache objects
222+
BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT: "mystorageaccount" # Required: Azure storage account name
223+
BUILDKITE_PLUGIN_AZURE_CACHE_PREFIX: "buildkite/cache"
224+
BUILDKITE_PLUGIN_AZURE_CACHE_QUIET: "true"
225+
BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE: "login" # Use Microsoft Entra ID authentication
226+
227+
steps:
228+
- label: ':nodejs: Install dependencies'
229+
command: npm ci
230+
plugins:
231+
- azure-login#v1.0.1: # Authenticate with managed identity first
232+
use-identity: true
233+
client-id: your-managed-identity-client-id
234+
- cache#v1.9.0:
235+
backend: azure
236+
path: node_modules
237+
manifest: package-lock.json
238+
restore: file
239+
save: file
240+
compression: zstd
241+
```
242+
243+
#### Example with Connection String
244+
245+
```yaml
246+
env:
247+
BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER: "my-cache-container" # Required: Azure container to store cache objects
248+
BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT: "mystorageaccount" # Required: Azure storage account name
249+
BUILDKITE_PLUGIN_AZURE_CACHE_PREFIX: "buildkite/cache"
250+
BUILDKITE_PLUGIN_AZURE_CACHE_QUIET: "true"
251+
252+
steps:
253+
- label: ':nodejs: Install dependencies'
254+
command: npm ci
255+
secrets:
256+
- AZURE_STORAGE_CONNECTION_STRING
257+
plugins:
258+
- cache#v1.9.0:
259+
backend: azure
260+
path: node_modules
261+
manifest: package-lock.json
262+
restore: file
263+
save: file
264+
compression: zstd
265+
```
266+
267+
#### Example with SAS Token
268+
269+
```yaml
270+
env:
271+
BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER: "my-cache-container" # Required: Azure container to store cache objects
272+
BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT: "mystorageaccount" # Required: Azure storage account name
273+
BUILDKITE_PLUGIN_AZURE_CACHE_PREFIX: "buildkite/cache"
274+
BUILDKITE_PLUGIN_AZURE_CACHE_QUIET: "true"
275+
276+
steps:
277+
- label: ':nodejs: Install dependencies'
278+
command: npm ci
279+
secrets:
280+
- AZURE_STORAGE_SAS_TOKEN
281+
plugins:
282+
- cache#v1.9.0:
283+
backend: azure
284+
path: node_modules
285+
manifest: package-lock.json
286+
restore: file
287+
save: file
288+
compression: zstd
289+
```
290+
167291
### `compression` (string)
168292

169293
Allows for the cached file/folder to be saved/restored as a single file. You will need to make sure to use the same compression when saving and restoring or it will cause a cache miss.

‎backends/cache_azure‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
#!/bin/bash
2+
3+
# Validate required configuration
4+
if [ -z "${BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER}" ]; then
5+
echo '+++ 🚨 Missing Azure container configuration'
6+
exit 1
7+
fi
8+
9+
if [ -z "${BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT}" ]; then
10+
echo '+++ 🚨 Missing Azure storage account configuration'
11+
exit 1
12+
fi
13+
14+
build_key() {
15+
if [ -n "${BUILDKITE_PLUGIN_AZURE_CACHE_PREFIX}" ]; then
16+
echo "${BUILDKITE_PLUGIN_AZURE_CACHE_PREFIX}/${1}"
17+
else
18+
echo "$1"
19+
fi
20+
}
21+
22+
az_cmd() {
23+
local subcommand="$1"
24+
shift
25+
26+
local cmd_args=(az storage blob "${subcommand}")
27+
cmd_args+=(--account-name "${BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT}")
28+
29+
if [[ "${BUILDKITE_PLUGIN_AZURE_CACHE_QUIET:-false}" =~ ^(true|on|1)$ ]]; then
30+
# Only upload and download support --no-progress
31+
if [ "${subcommand}" = "upload" ] || [ "${subcommand}" = "download" ]; then
32+
cmd_args+=(--no-progress)
33+
fi
34+
cmd_args+=(--only-show-errors)
35+
fi
36+
37+
if [ -n "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}" ]; then
38+
cmd_args+=(--auth-mode "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}")
39+
40+
# When using key auth mode, explicitly pass the account key
41+
if [ "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}" = "key" ] && [ -n "${AZURE_STORAGE_KEY}" ]; then
42+
cmd_args+=(--account-key "${AZURE_STORAGE_KEY}")
43+
fi
44+
fi
45+
46+
# When quiet mode is enabled, suppress stdout (progress bars, JSON output)
47+
# but preserve stderr (error messages)
48+
if [[ "${BUILDKITE_PLUGIN_AZURE_CACHE_QUIET:-false}" =~ ^(true|on|1)$ ]]; then
49+
"${cmd_args[@]}" "$@" >/dev/null
50+
else
51+
"${cmd_args[@]}" "$@"
52+
fi
53+
}
54+
55+
azure_exists() {
56+
local key="$1"
57+
local blob_name
58+
blob_name="$(build_key "${key}")"
59+
60+
# Try to check if exact blob exists
61+
if az_cmd show \
62+
--container-name "${BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER}" \
63+
--name "${blob_name}" &>/dev/null; then
64+
return 0
65+
fi
66+
67+
# Check if it's a directory-like prefix (with trailing slash)
68+
local list_args=(az storage blob list)
69+
list_args+=(--container-name "${BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER}")
70+
list_args+=(--account-name "${BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT}")
71+
list_args+=(--prefix "${blob_name}/")
72+
list_args+=(--num-results 1)
73+
list_args+=(--query '[0].name')
74+
list_args+=(--output tsv)
75+
76+
# Note: list command does not support --no-progress
77+
if [[ "${BUILDKITE_PLUGIN_AZURE_CACHE_QUIET:-false}" =~ ^(true|on|1)$ ]]; then
78+
list_args+=(--only-show-errors)
79+
fi
80+
81+
if [ -n "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}" ]; then
82+
list_args+=(--auth-mode "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}")
83+
84+
# When using key auth mode, explicitly pass the account key
85+
if [ "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}" = "key" ] && [ -n "${AZURE_STORAGE_KEY}" ]; then
86+
list_args+=(--account-key "${AZURE_STORAGE_KEY}")
87+
fi
88+
fi
89+
90+
local result
91+
result=$("${list_args[@]}" 2>/dev/null)
92+
93+
[ -n "${result}" ]
94+
}
95+
96+
restore_cache() {
97+
local from="$1"
98+
local to="$2"
99+
local blob_name
100+
blob_name="$(build_key "${from}")"
101+
102+
# Check if it's a single blob or a directory
103+
if az_cmd show \
104+
--container-name "${BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER}" \
105+
--name "${blob_name}" &>/dev/null; then
106+
# Single file - use download
107+
az_cmd download \
108+
--container-name "${BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER}" \
109+
--name "${blob_name}" \
110+
--file "${to}"
111+
else
112+
# Directory - use download-batch
113+
local batch_args=(az storage blob download-batch)
114+
batch_args+=(--account-name "${BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT}")
115+
batch_args+=(--source "${BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER}")
116+
batch_args+=(--destination "${to}")
117+
batch_args+=(--pattern "${blob_name}/*")
118+
119+
if [[ "${BUILDKITE_PLUGIN_AZURE_CACHE_QUIET:-false}" =~ ^(true|on|1)$ ]]; then
120+
batch_args+=(--no-progress)
121+
batch_args+=(--only-show-errors)
122+
fi
123+
124+
if [ -n "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}" ]; then
125+
batch_args+=(--auth-mode "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}")
126+
127+
# When using key auth mode, explicitly pass the account key
128+
if [ "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}" = "key" ] && [ -n "${AZURE_STORAGE_KEY}" ]; then
129+
batch_args+=(--account-key "${AZURE_STORAGE_KEY}")
130+
fi
131+
fi
132+
133+
# When quiet mode is enabled, suppress stdout but preserve stderr
134+
if [[ "${BUILDKITE_PLUGIN_AZURE_CACHE_QUIET:-false}" =~ ^(true|on|1)$ ]]; then
135+
"${batch_args[@]}" >/dev/null
136+
else
137+
"${batch_args[@]}"
138+
fi
139+
fi
140+
}
141+
142+
save_cache() {
143+
local to="$1"
144+
local from="$2"
145+
local blob_name
146+
blob_name="$(build_key "${to}")"
147+
148+
if [ ! -e "${from}" ]; then
149+
echo "+++ 🚨 Cache source path does not exist: ${from}" >&2
150+
return 1
151+
fi
152+
153+
if [ -f "${from}" ]; then
154+
# Single file - use upload
155+
az_cmd upload \
156+
--container-name "${BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER}" \
157+
--name "${blob_name}" \
158+
--file "${from}" \
159+
--overwrite
160+
else
161+
# Directory - use upload-batch
162+
local batch_args=(az storage blob upload-batch)
163+
batch_args+=(--account-name "${BUILDKITE_PLUGIN_AZURE_CACHE_ACCOUNT}")
164+
batch_args+=(--destination "${BUILDKITE_PLUGIN_AZURE_CACHE_CONTAINER}")
165+
batch_args+=(--source "${from}")
166+
batch_args+=(--destination-path "${blob_name}")
167+
batch_args+=(--overwrite)
168+
169+
if [[ "${BUILDKITE_PLUGIN_AZURE_CACHE_QUIET:-false}" =~ ^(true|on|1)$ ]]; then
170+
batch_args+=(--no-progress)
171+
batch_args+=(--only-show-errors)
172+
fi
173+
174+
if [ -n "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}" ]; then
175+
batch_args+=(--auth-mode "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}")
176+
177+
# When using key auth mode, explicitly pass the account key
178+
if [ "${BUILDKITE_PLUGIN_AZURE_CACHE_AUTH_MODE}" = "key" ] && [ -n "${AZURE_STORAGE_KEY}" ]; then
179+
batch_args+=(--account-key "${AZURE_STORAGE_KEY}")
180+
fi
181+
fi
182+
183+
# When quiet mode is enabled, suppress stdout but preserve stderr
184+
if [[ "${BUILDKITE_PLUGIN_AZURE_CACHE_QUIET:-false}" =~ ^(true|on|1)$ ]]; then
185+
"${batch_args[@]}" >/dev/null
186+
else
187+
"${batch_args[@]}"
188+
fi
189+
fi
190+
}
191+
192+
exists_cache() {
193+
if [ -z "$1" ]; then exit 1; fi
194+
azure_exists "$1"
195+
}
196+
197+
OPCODE="$1"
198+
shift
199+
200+
if [ "$OPCODE" = 'exists' ]; then
201+
exists_cache "$@"
202+
elif [ "$OPCODE" = 'get' ]; then
203+
restore_cache "$@"
204+
elif [ "$OPCODE" = 'save' ]; then
205+
save_cache "$@"
206+
else
207+
exit 255
208+
fi

0 commit comments

Comments
 (0)