Support warm-up of loader caches by jakob-keller · Pull Request #1462 · aio-libs/aiobotocore

jakob-keller · 2026-01-02T21:00:13Z

Description of Change

Support warm-up of loader caches

Assumptions

Replace this text with any assumptions made (if any)

Checklist for All Submissions

Checklist when updating botocore and/or aiohttp versions

I have read and followed CONTRIBUTING.rst
I have updated test_patches.py where/if appropriate (also check if no changes necessary)
I have added URL to diff: boto/botocore@[CURRENT_BOTO_VERSION_TAG]...[NEW_BOTO_VERSION_TAG]

gemini-code-assist · 2026-01-02T21:00:16Z

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

codecov · 2026-01-02T21:07:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.92%. Comparing base (d0ac8b9) to head (3fad65c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1462      +/-   ##
==========================================
+ Coverage   93.86%   93.92%   +0.05%     
==========================================
  Files          77       77              
  Lines        8218     8298      +80     
==========================================
+ Hits         7714     7794      +80     
  Misses        504      504

Flag	Coverage Δ
no-httpx	`90.81% <100.00%> (+0.08%)`	⬆️
os-ubuntu-24.04	`93.92% <100.00%> (+0.05%)`	⬆️
os-ubuntu-24.04-arm	`91.98% <100.00%> (+0.07%)`	⬆️
python-3.10	`91.95% <100.00%> (+0.07%)`	⬆️
python-3.11	`91.95% <100.00%> (+0.07%)`	⬆️
python-3.12	`91.95% <100.00%> (+0.07%)`	⬆️
python-3.13	`91.95% <100.00%> (+0.07%)`	⬆️
python-3.14	`93.89% <100.00%> (+0.05%)`	⬆️
python-3.9	`91.96% <100.00%> (+0.07%)`	⬆️
unittests	`93.92% <100.00%> (+0.05%)`	⬆️
with-awscrt	`93.57% <100.00%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

webknjaz · 2026-01-18T21:26:07Z

I can't really review this in-depth, just left a few infra/testing comments.

jakob-keller · 2026-01-18T21:32:23Z

I can't really review this in-depth, just left a few infra/testing comments.

Fair enough and thank you anyway!

patrickvorgers · 2026-01-20T20:13:36Z

I tried to validate this in Home Assistant (IDrive e2 backup provider), but Home Assistant is currently pinned to aiobotocore 2.x. Would it be possible to make this warm-up API available (or backported) on the 2.x line as well, so downstream projects pinned to 2.x can adopt and test it?

jakob-keller · 2026-01-20T20:22:22Z

Would it be possible to make this warm-up API available (or backported) on the 2.x line as well, so downstream projects pinned to 2.x can adopt and test it?

We typically don't provide backports and there are no plans to publish additional 2.x releases.

It should be straightforward to experiment with the code snippets I posted previously or the proposed changes in this PR. That should work with any version of aiobotocore, including 2.x releases.

patrickvorgers · 2026-01-20T20:24:56Z

Home Assistant is currently using aiobotocore 2.21.1 so it is probably good anyhow to check whether it is possible to upgrade to the latest version.

jakob-keller · 2026-01-20T20:27:45Z

Home Assistant is currently using aiobotocore 2.21.1 so it is probably good anyhow to check whether it is possible to upgrade to the latest version.

Possibly, but that won't change anything: You still need to duplicate my proposed cache warming code, if you want to test it. So far, it's only contained in this pending PR and we are awaiting community feedback before we agree on a way to move forward.

patrickvorgers · 2026-01-20T21:11:35Z

I applied the change manually to aiobotocore 2.21.1 and preload the caches.

session = AioSession()
await hass.async_add_executor_job(session.warm_up_loader_caches, "s3")

Some detections like the listdir are gone now but a SSL one remains:

2026-01-20 21:02:16.696 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to load_verify_locations with args (<ssl.SSLContext object at 0x7f191de3f110>, '/home/vscode/.local/ha-venv/lib/python3.13/site-packages/certifi/cacert.pem', None, None) inside the event loop by integration 'idrive_e2' at homeassistant/components/idrive_e2/__init__.py, line 46: await cast(Any, client).head_bucket(Bucket=entry.data[CONF_BUCKET]) (offender: /home/vscode/.local/ha-venv/lib/python3.13/site-packages/aiobotocore/httpsession.py, line 165: ssl_context.load_verify_locations(ca_certs, None, None)), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+idrive_e2%22
For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#load_verify_locations
Traceback (most recent call last):
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 71, in <module>
    cli.main()
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 508, in main
    run()
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 391, in run_module
    run_module_as_main(options.target, alter_argv=True)
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 228, in _run_module_as_main
    return _run_code(code, main_globals, None, "__main__", mod_spec)
  File "/home/vscode/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
    exec(code, run_globals)
  File "/workspaces/core/homeassistant/__main__.py", line 229, in <module>
    sys.exit(main())
  File "/workspaces/core/homeassistant/__main__.py", line 215, in main
    exit_code = runner.run(runtime_conf)
  File "/workspaces/core/homeassistant/runner.py", line 289, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 712, in run_until_complete
    self.run_forever()
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 683, in run_forever
    self._run_once()
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 2042, in _run_once
    handle._run()
  File "/home/vscode/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu/lib/python3.13/asyncio/events.py", line 89, in _run
    self._context.run(self._callback, *self._args)
  File "/workspaces/core/homeassistant/config_entries.py", line 928, in async_setup_locked
    await self.async_setup(hass, integration=integration)
  File "/workspaces/core/homeassistant/config_entries.py", line 672, in async_setup
    await self.__async_setup_with_context(hass, integration)
  File "/workspaces/core/homeassistant/config_entries.py", line 762, in __async_setup_with_context
    result = await component.async_setup_entry(hass, self)
  File "/workspaces/core/homeassistant/components/idrive_e2/__init__.py", line 46, in async_setup_entry
    await cast(Any, client).head_bucket(Bucket=entry.data[CONF_BUCKET])

jakob-keller · 2026-01-20T21:15:27Z

I applied the change manually to aiobotocore 2.21.1 and preload the caches.
session = AioSession()
await hass.async_add_executor_job(session.warm_up_loader_caches, "s3")
Some detections like the listdir are gone now but a SSL one remains:

Sounds to me like a confirmation that the proposed approach works in the HA use case.

patrickvorgers · 2026-01-21T08:26:11Z

I applied the change manually to aiobotocore 2.21.1 and preload the caches.
session = AioSession()
await hass.async_add_executor_job(session.warm_up_loader_caches, "s3")
Some detections like the listdir are gone now but a SSL one remains:
Sounds to me like a confirmation that the proposed approach works in the HA use case.

Is there also a way to get rid of the SSL one?

ssl_context.load_verify_locations(ca_certs, None, None)

jakob-keller · 2026-01-21T08:27:49Z

Is there also a way to get rid of the SSL one?

Glancing at the stacktrace you shared, I don't see how that would be related to aiobotocore.

patrickvorgers · 2026-01-21T08:30:52Z

Is there also a way to get rid of the SSL one?

Glancing at the stacktrace you shared, I don't see how that would be related to aiobotocore.

It seems to be triggered by:

await cast(Any, client).head_bucket(Bucket=entry.data[CONF_BUCKET])

Which is part of aiobotocore.

jakob-keller · 2026-01-21T10:39:46Z

Is there also a way to get rid of the SSL one?

Glancing at the stacktrace you shared, I don't see how that would be related to aiobotocore.

It seems to be triggered by:
await cast(Any, client).head_bucket(Bucket=entry.data[CONF_BUCKET])
Which is part of aiobotocore.

Unblocking setup of the SSL context is another can of worms. This is out of scope of this PR. Please create a separate issue.

thehesiod · 2026-01-22T14:37:06Z

the SSL context thing sounds like it could be easily handled by my version where we have an executor. sorry haven't had a chance to review the comments across all the issues yet in case this was already discussed, will try to catch up soon

patrickvorgers · 2026-01-22T19:40:45Z

the SSL context thing sounds like it could be easily handled by my version where we have an executor. sorry haven't had a chance to review the comments across all the issues yet in case this was already discussed, will try to catch up soon

It would be great if there is already a solution in place that could work.
Any ideas on how to verify this and move this forward?

thehesiod · 2026-03-18T19:50:17Z

ok I'll try to get AI to enhance mine to support the ssl case hopefully tonight. Sorry been crazy busy last couple months. Lots of plans for aiobotocore now with AI!

patrickvorgers · 2026-03-18T22:02:13Z

ok I'll try to get AI to enhance mine to support the ssl case hopefully tonight. Sorry been crazy busy last couple months. Lots of plans for aiobotocore now with AI!

That would be great, thank you for looking into it!

thehesiod · 2026-04-07T08:19:03Z

@claude can you read the comments in this PR regarding SSL context loading issue by patrickvorgers, would it be easy to be handled in #1451 ?

claude

Code review

Bug: raises for services without optional model types (confidence: 95)

The warm_up_loader_caches method calls load_service_model for waiters-2, paginators-1, and examples-1 without catching DataNotFoundError. Many services (e.g. iot, accessanalyzer, account) don't have a waiters-2 file, causing UnknownServiceError (a subclass of DataNotFoundError) to be raised and failing client creation when warm_up_loader_caches=True.

Verified locally:

>>> from botocore.loaders import Loader
>>> Loader().load_service_model('iot', 'waiters-2')
# raises UnknownServiceError

Botocore's own client.py wraps these calls in try/except DataNotFoundError (see _get_waiter_config). The warm-up method should do the same:

# from session.py
loader.load_data_with_path('endpoints')
loader.load_data('sdk-default-configuration')
try:
    loader.load_service_model(service_name, 'waiters-2', api_version)
except DataNotFoundError:
    pass
try:
    loader.load_service_model(service_name, 'paginators-1', api_version)
except DataNotFoundError:
    pass
loader.load_service_model(
    service_name, type_name='service-2', api_version=api_version
)
loader.list_available_services(type_name='service-2')

# from client.py
loader.load_data('partitions')
loader.load_service_model(
    service_name, 'service-2', api_version=api_version
)
loader.load_service_model(
    service_name, 'endpoint-rule-set-1', api_version=api_version
)
loader.load_data('_retry')

# from docs/service.py
try:
    loader.load_service_model(service_name, 'examples-1', api_version)
except DataNotFoundError:
    pass

You'll also need to import DataNotFoundError from botocore.exceptions, and add a test case for a service that lacks waiters-2 (e.g. iot or accessanalyzer).

🤖 Generated with Claude Code

jakob-keller added the enhancement New feature or request label Jan 2, 2026

jakob-keller force-pushed the warm-up-loader-caches branch from 726bb38 to eca8210 Compare January 2, 2026 21:03

jakob-keller mentioned this pull request Jan 2, 2026

Add support for executor configuration in AioConfig for file loading #1451

Open

10 tasks

jakob-keller commented Jan 2, 2026

View reviewed changes

Comment thread aiobotocore/session.py

jakob-keller mentioned this pull request Jan 2, 2026

Detected blocking call inside the event loop #1199

Open

6 tasks

jakob-keller force-pushed the warm-up-loader-caches branch 2 times, most recently from d2bc4da to 74d9cef Compare January 3, 2026 00:27

jakob-keller added the bug Something isn't working label Jan 3, 2026

jakob-keller force-pushed the warm-up-loader-caches branch 2 times, most recently from 7eb8690 to c60df2e Compare January 3, 2026 03:36

jakob-keller marked this pull request as ready for review January 3, 2026 03:40

jakob-keller requested review from bdraco, thehesiod and webknjaz January 3, 2026 03:42

patrickvorgers mentioned this pull request Jan 18, 2026

IDrive e2 backup provider home-assistant/core#144910

Merged

19 tasks

webknjaz reviewed Jan 18, 2026

View reviewed changes

Comment thread aiobotocore/session.py

webknjaz reviewed Jan 18, 2026

View reviewed changes

Comment thread tests/test_session.py Outdated

jakob-keller force-pushed the warm-up-loader-caches branch 3 times, most recently from 53c2115 to 95b54a2 Compare January 20, 2026 16:43

patrickvorgers mentioned this pull request Jan 21, 2026

Unblocking creation of SSL context #1469

Open

6 tasks

jakob-keller force-pushed the warm-up-loader-caches branch 2 times, most recently from 42d671d to bbc0134 Compare February 5, 2026 10:04

owine mentioned this pull request Apr 2, 2026

IDrive e2 blocks event loop with synchronous listdir/open calls during setup home-assistant/core#167242

Open

claude bot reviewed Apr 7, 2026

View reviewed changes

This comment was marked as duplicate.

Sign in to view

Support warm-up of loader caches

78ef1ff

jakob-keller force-pushed the warm-up-loader-caches branch from bbc0134 to a9859d6 Compare April 7, 2026 09:51

Generalize AioSession.warm_up_loader_caches()

3fad65c

jakob-keller force-pushed the warm-up-loader-caches branch from a9859d6 to 3fad65c Compare April 7, 2026 09:54

Uh oh!

Conversation

jakob-keller commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Change

Assumptions

Checklist for All Submissions

Checklist when updating botocore and/or aiohttp versions

Uh oh!

gemini-code-assist bot commented Jan 2, 2026

Uh oh!

codecov bot commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

webknjaz commented Jan 18, 2026

Uh oh!

jakob-keller commented Jan 18, 2026

Uh oh!

patrickvorgers commented Jan 20, 2026

Uh oh!

jakob-keller commented Jan 20, 2026

Uh oh!

patrickvorgers commented Jan 20, 2026

Uh oh!

jakob-keller commented Jan 20, 2026

Uh oh!

patrickvorgers commented Jan 20, 2026

Uh oh!

jakob-keller commented Jan 20, 2026

Uh oh!

patrickvorgers commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakob-keller commented Jan 21, 2026

Uh oh!

patrickvorgers commented Jan 21, 2026

Uh oh!

jakob-keller commented Jan 21, 2026

Uh oh!

thehesiod commented Jan 22, 2026

Uh oh!

patrickvorgers commented Jan 22, 2026

Uh oh!

thehesiod commented Mar 18, 2026

Uh oh!

patrickvorgers commented Mar 18, 2026

Uh oh!

thehesiod commented Apr 7, 2026

Uh oh!

claude bot left a comment

Choose a reason for hiding this comment

Code review

Uh oh!

This comment was marked as duplicate.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jakob-keller commented Jan 2, 2026 •

edited

Loading

codecov bot commented Jan 2, 2026 •

edited

Loading

patrickvorgers commented Jan 21, 2026 •

edited

Loading