Using azd to manage infra for E2E Python tests#363
Using azd to manage infra for E2E Python tests#363rodrigobr-msft wants to merge 1 commit intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR migrates E2E/integration test infrastructure management for the Python SDK to azd + Bicep under dev/testing/python-sdk-tests, and removes the legacy cross-SDK test suite in preparation for moving it to the main repo.
Changes:
- Added an
azdproject (Bicep + hooks + scripts) to provision App Registration, Azure Bot, and Key Vault and populate test.env. - Added local Docker support for running
python-sdk-testsin a multi-runtime test container. - Removed
dev/testing/cross-sdk-testsimplementation and updated testing docs accordingly.
Reviewed changes
Copilot reviewed 66 out of 79 changed files in this pull request and generated 20 comments.
Show a summary per file
| File | Description |
|---|---|
| dev/testing/README.md | Updates test directory overview and adds instructions for local/azd/Docker runs. |
| dev/testing/python-sdk-tests/README.md | Expanded setup + run documentation for the Python integration tests. |
| dev/testing/python-sdk-tests/pyproject.toml | Introduces a uv-managed test project with editable path sources to local SDK packages. |
| dev/testing/python-sdk-tests/main.py | Adds project scaffold entrypoint (currently placeholder). |
| dev/testing/python-sdk-tests/env.TEMPLATE | Switches to template placeholders for injected infra values. |
| dev/testing/python-sdk-tests/azure.yaml | Defines azd infra configuration and Windows hook scripts. |
| dev/testing/python-sdk-tests/.gitignore | Ignores generated infra output file. |
| dev/testing/python-sdk-tests/tests.Dockerfile | Adds multi-runtime Docker image for running tests via uv run pytest. |
| dev/testing/python-sdk-tests/scripts/run_local.ps1 | Adds a Docker-based local runner script. |
| dev/testing/python-sdk-tests/scripts/postprovision.ps1 | Adds azd postprovision hook to write .infra.json and inject config. |
| dev/testing/python-sdk-tests/scripts/predown.ps1 | Adds azd predown hook to delete Entra App Registration to prevent orphaning. |
| dev/testing/python-sdk-tests/scripts/provision.ps1 | Adds standalone provisioning script (non-azd) for local infra + config injection. |
| dev/testing/python-sdk-tests/scripts/inject_config.py | Adds generic ${{ path }} template substitution utility for config injection. |
| dev/testing/python-sdk-tests/run_tests.ps1 | Removes old PowerShell test runner. |
| dev/testing/python-sdk-tests/infra/bicepconfig.json | Enables Bicep extensibility and Microsoft Graph extension configuration. |
| dev/testing/python-sdk-tests/infra/main.bicep | New top-level Bicep deployment for App Registration, Bot Service, and Key Vault. |
| dev/testing/python-sdk-tests/infra/main.bicepparam | Parameter file using environment variables for defaults. |
| dev/testing/python-sdk-tests/infra/modules/app-registration.bicep | Graph-based App Registration + Service Principal module. |
| dev/testing/python-sdk-tests/infra/modules/azure-bot.bicep | Azure Bot Service registration module. |
| dev/testing/python-sdk-tests/infra/modules/key-vault.bicep | Key Vault module with RBAC role assignment for secret write access. |
| dev/testing/microsoft-agents-testing/pyproject.toml | Adds pydantic-settings dependency to the testing framework package. |
| dev/testing/cross-sdk-tests/Program.cs | Adds a minimal C# file (comment-only) to satisfy CodeQL expectations. |
| dev/testing/cross-sdk-tests/pytest.ini | Removes legacy cross-SDK pytest configuration. |
| dev/testing/cross-sdk-tests/env.TEMPLATE | Removes legacy cross-SDK env template. |
| dev/testing/cross-sdk-tests/.gitignore | Removes legacy cross-SDK gitignore. |
| dev/testing/cross-sdk-tests/tests/basic/test_quickstart.py | Removes legacy cross-SDK quickstart tests. |
| dev/testing/cross-sdk-tests/tests/core/test_basic_agent_base.py | Removes legacy shared base test class. |
| dev/testing/cross-sdk-tests/tests/core/test_directline.py | Removes legacy Direct Line tests. |
| dev/testing/cross-sdk-tests/tests/core/test_msteams.py | Removes legacy Teams tests. |
| dev/testing/cross-sdk-tests/tests/core/test_webchat.py | Removes legacy WebChat tests. |
| dev/testing/cross-sdk-tests/tests/telemetry/test_basic_telemetry.py | Removes legacy telemetry placeholder tests. |
| dev/testing/cross-sdk-tests/tests/_common/init.py | Removes legacy cross-SDK test helpers export module. |
| dev/testing/cross-sdk-tests/tests/_common/constants.py | Removes legacy constants for cross-SDK tests. |
| dev/testing/cross-sdk-tests/tests/_common/source_scenario.py | Removes legacy source-scenario runner. |
| dev/testing/cross-sdk-tests/tests/_common/types.py | Removes legacy SDKVersion enum. |
| dev/testing/cross-sdk-tests/tests/_common/utils.py | Removes legacy scenario/path utilities. |
| dev/testing/cross-sdk-tests/agents/quickstart/README.md | Removes legacy quickstart agent docs. |
| dev/testing/cross-sdk-tests/agents/quickstart/python/_run_agent.ps1 | Removes legacy Python agent runner script. |
| dev/testing/cross-sdk-tests/agents/quickstart/python/requirements.txt | Removes legacy Python agent requirements. |
| dev/testing/cross-sdk-tests/agents/quickstart/python/src/agent.py | Removes legacy Python quickstart agent implementation. |
| dev/testing/cross-sdk-tests/agents/quickstart/python/src/main.py | Removes legacy Python quickstart entrypoint. |
| dev/testing/cross-sdk-tests/agents/quickstart/python/src/start_server.py | Removes legacy Python quickstart server host. |
| dev/testing/cross-sdk-tests/agents/quickstart/net/_run_agent.ps1 | Removes legacy .NET agent runner script. |
| dev/testing/cross-sdk-tests/agents/quickstart/net/AspNetExtensions.cs | Removes legacy .NET hosting/auth helpers. |
| dev/testing/cross-sdk-tests/agents/quickstart/net/MyAgent.cs | Removes legacy .NET quickstart agent. |
| dev/testing/cross-sdk-tests/agents/quickstart/net/Program.cs | Removes legacy .NET app host. |
| dev/testing/cross-sdk-tests/agents/quickstart/net/Quickstart.csproj | Removes legacy .NET project file. |
| dev/testing/cross-sdk-tests/agents/quickstart/js/_run_agent.ps1 | Removes legacy JS agent runner script. |
| dev/testing/cross-sdk-tests/agents/quickstart/js/env.TEMPLATE | Removes legacy JS env template. |
| dev/testing/cross-sdk-tests/agents/quickstart/js/package.json | Removes legacy JS agent package definition. |
| dev/testing/cross-sdk-tests/agents/quickstart/js/src/index.ts | Removes legacy JS quickstart agent implementation. |
| dev/testing/cross-sdk-tests/agents/quickstart/js/tsconfig.json | Removes legacy JS TS config. |
| dev/testing/cross-sdk-tests/agents/core-agent/README.md | Removes legacy core-agent docs. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/env.TEMPLATE | Removes legacy core-agent env template. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/pre_requirements.txt | Removes legacy core-agent prereqs. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/README.md | Removes legacy core-agent Python README. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/requirements.txt | Removes legacy core-agent requirements. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/src/agent.py | Removes legacy core-agent main logic. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/src/app.py | Removes legacy core-agent aiohttp app host. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/src/config.py | Removes legacy core-agent config class. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/src/weather/agents/weather_forecast_agent.py | Removes legacy weather agent implementation. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/src/weather/plugins/init.py | Removes legacy plugin exports. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/src/weather/plugins/adaptive_card_plugin.py | Removes legacy adaptive card plugin. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/src/weather/plugins/date_time_plugin.py | Removes legacy datetime plugin. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/src/weather/plugins/weather_forecast.py | Removes legacy forecast model. |
| dev/testing/cross-sdk-tests/agents/core-agent/python/src/weather/plugins/weather_forecast_plugin.py | Removes legacy forecast plugin. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if args.template: | ||
| _process_file(args.template, args.output, variables) | ||
| else: | ||
| scenario_dir = Path(__file__).parent.parent | ||
| _discover_and_process(scenario_dir, variables) |
There was a problem hiding this comment.
--scenario discovery mode ignores the provided scenario name: when args.template is not set, scenario_dir is hard-coded to Path(__file__).parent.parent and args.scenario is never used. This makes the CLI help misleading and can cause template substitution to run over unintended directories. Use args.scenario to construct the intended scenario root (or remove the flag if discovery should always scan the entire tree).
| $EnvDir = (Resolve-Path "$PSScriptRoot/..").Path | ||
| $RepoRoot = (Resolve-Path "$PSScriptRoot/..").Path |
There was a problem hiding this comment.
$RepoRoot is set to the same path as $EnvDir (the python-sdk-tests/ directory). Because pyproject.toml uses editable path sources that point outside this folder (e.g., ../microsoft-agents-testing, ../../../libraries/...), mounting only this directory at /repo will make those sources unavailable inside the container and uv dependency resolution will fail. Resolve $RepoRoot to the actual repository root (and adjust Docker WORKDIR / docker run accordingly).
| # Build: docker build -f environments/local/tests.Dockerfile -t agents-test . | ||
| # Run: docker run --rm -v "${PWD}:/repo" agents-test environments/local/tests/ | ||
|
|
There was a problem hiding this comment.
The Dockerfile header’s build/run examples reference environments/local/tests.Dockerfile and environments/local/tests/, but this file lives at dev/testing/python-sdk-tests/tests.Dockerfile. These instructions will be incorrect for anyone following them; update the paths to match the repository layout.
| WORKDIR /repo | ||
| # Keep the venv inside the container, not on the mounted Windows volume. | ||
| # Without this, uv tries to recreate /repo/.venv (Windows-style Scripts/ dir) | ||
| # and hits cross-OS I/O errors. | ||
| ENV UV_PROJECT_ENVIRONMENT=/venv | ||
| ENTRYPOINT ["uv", "run", "pytest"] |
There was a problem hiding this comment.
WORKDIR /repo assumes the mounted directory contains the python-sdk-tests pyproject, but run_local.ps1 (and the file header) describe mounting the whole repo at /repo. If the repo root is mounted, uv run pytest will not find dev/testing/python-sdk-tests/pyproject.toml from /repo. Consider setting WORKDIR to /repo/dev/testing/python-sdk-tests (and keep the volume mount as the repo root) so editable path sources resolve correctly.
| # --- 2. Get deployer principal ID (Key Vault Secrets Officer assignment) --- | ||
| $DeployerPrincipalId = az ad signed-in-user show --query id --output tsv 2>$null | ||
| if (-not $DeployerPrincipalId) { | ||
| $DeployerPrincipalId = az account show --query user.name --output tsv | ||
| } |
There was a problem hiding this comment.
$DeployerPrincipalId fallback uses az account show --query user.name, which returns a UPN/appId-like value, not the Entra objectId required for RBAC role assignments. This will break Key Vault role assignment when az ad signed-in-user show is unavailable (e.g., service principal auth). Retrieve the correct objectId (e.g., via az ad sp show --id ... --query id for SPs) before passing it to Bicep.
| # Self-contained azd config for the LOCAL test environment. | ||
| # Run `azd provision` from this directory (environments/local/). | ||
| # |
There was a problem hiding this comment.
The azd project comments mention running from environments/local/, but this azure.yaml is located in dev/testing/python-sdk-tests/. Update the comment so users run azd from the correct directory.
| description = "Add your description here" | ||
| readme = "README.md" | ||
| requires-python = ">=3.13" |
There was a problem hiding this comment.
description = "Add your description here" is still the template placeholder, and requires-python = ">=3.13" is stricter than the rest of the repo (packages appear to support 3.10+). If the tests don’t require 3.13-specific features, consider lowering requires-python to match the SDK and replacing the placeholder description to avoid confusion.
| def main(): | ||
| print("Hello from python-sdk-tests!") |
There was a problem hiding this comment.
This file looks like the default uv init scaffold and doesn’t appear to be used by the test suite. Keeping an unused main.py in the test project adds noise and can confuse readers/tools; consider removing it if it’s not serving a purpose.
| @@ -0,0 +1 @@ | |||
| .infra.json No newline at end of file | |||
There was a problem hiding this comment.
Only .infra.json is ignored here, but scripts/provision.ps1 writes infra-outputs.json. If provision.ps1 is still intended to be used, add infra-outputs.json (and possibly other generated config artifacts) to this ignore list or standardize on a single outputs filename across scripts.
| "microsoft-agents-authentication-msal", | ||
| "pydantic", | ||
| "pydantic-settings", | ||
| "pytest", | ||
| "pytest-aiohttp", |
There was a problem hiding this comment.
pydantic-settings is added as a runtime dependency, but there are no imports/usages of it in this package. If it’s not required, removing it will keep the dependency surface smaller; if it is required, consider adding a reference (or a comment) showing where it’s used.
This PR also removes the old cross-SDK testing suite, as it will be moved to the main repo.