Race condition in credential management when reusing same session across tasks

[leoleoasd]

Describe the bug
A clear and concise description of what the bug is.

I'm reusing same session in multiple tasks and concurrently launching them. my aws config looks like:

[profile a]
role_arn = xxx
source_profile= b

[profile b]
role_arn = xxx
credential_source = Ec2InstanceMetadata

and got this error:

Infinite loop in credential configuration detected. Attempting to load from profile yuxuanlu which has already been visited. Visited profiles: ['a', 'b'];

My suspicion is first task is accessing a then b, and marking b as visited, and second task encountered error when visiting b as well. Maybe a global lock is needed for credential management. I added an STS call before launching my tasks (so credentials are fetched and cached) and no longer seeing the issue:

async with self.boto_session.client("sts", region_name=self.region) as sts:
    identity = await sts.get_caller_identity()
    print(f"Caller identity: {identity}")

Checklist

• I have reproduced in environment where pip check passes without errors
• I have provided pip freeze results
• I have provided sample code or detailed way to reproduce
• I have tried the same code in botocore to ensure this is an aiobotocore specific issue
• I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
• I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results

Environment:

• Python Version: 3.13.2
• OS name and version: linux

Additional context
Add any other context about the problem here.

[thehesiod]

@claude can you research this issue

[claude]

Research Findings: Race Condition in AioAssumeRoleProvider

Root Cause

The bug is in AioAssumeRoleProvider in aiobotocore/credentials.py. The _visited_profiles list (inherited from botocore's AssumeRoleProvider) is an instance variable initialized once in __init__:

# botocore/credentials.py (AssumeRoleProvider.__init__)
self._visited_profiles = [self._profile_name]

In botocore this is fine because load() is synchronous — each call completes atomically. But in aiobotocore, load() is async def with await points. When multiple tasks concurrently resolve credentials using the same session (and thus the same AioAssumeRoleProvider instance), they mutate the shared _visited_profiles list simultaneously:

Task 1                                    Task 2
──────────────────────────────────────────────────────────────────
load() called
_visited_profiles = ['a']
append 'b' → ['a', 'b']
await _resolve_credentials_from_profile('b')   ← suspends here

                                          load() called
                                          _validate_source_profile('b')
                                          'b' in _visited_profiles? → TRUE
                                          → InfiniteLoopConfigError ❌
                                          (but Task 2 never actually visited 'b')

The mutation happens at credentials.py:840:

async def _resolve_source_credentials(self, role_config, profile_name):
    source_profile = role_config['source_profile']
    self._visited_profiles.append(source_profile)  # ← shared mutable state
    return await self._resolve_credentials_from_profile(source_profile)

Why the workaround works

The user's workaround (making an STS call before launching concurrent tasks) populates the credential cache. Subsequent calls hit the cache and skip the _visited_profiles resolution path entirely.

Proposed Fix

Reset _visited_profiles at the start of each load() call in AioAssumeRoleProvider:

async def load(self):
    self._visited_profiles = [self._profile_name]  # reset per load call
    self._loaded_config = self._load_config()
    profiles = self._loaded_config.get('profiles', {})
    profile = profiles.get(self._profile_name, {})
    if self._has_assume_role_config_vars(profile):
        return await self._load_creds_via_assume_role(self._profile_name)

This ensures each concurrent load() call has its own logical view of _visited_profiles. Cycle detection within a single chain resolution still works correctly because the recursive calls to _load_creds_via_assume_role → _resolve_source_credentials all happen in the same task's execution path and still append to the same list. Two concurrent tasks no longer share state.

An alternative (and more robust) approach would be to use contextvars.ContextVar so that each async task has truly isolated visited-profile tracking, but the simple reset approach should be sufficient given that _visited_profiles is only meaningful within a single top-level load() invocation.

[thehesiod]

@claude is this an issue unique to aiobotocore, or could it also happen when using botocore directly?

[thehesiod]

@jakob-keller putting our agent to work ;)

[claude]

Analysis: Is this unique to aiobotocore?

Short answer: The same race condition is theoretically possible in botocore with multi-threaded usage, but aiobotocore makes it far more likely to occur in practice.

Why it's practically unique to aiobotocore

In synchronous botocore, load() is called once per provider instance. Python's GIL prevents true parallelism, but even with multiple threads, real I/O (STS, IMDS) releases the GIL. So if two threads share the same session and both trigger credential resolution before caching kicks in, they could race on _visited_profiles in the same way.

However, sharing a botocore session across threads is unusual and generally warned against. The typical botocore usage is single-threaded, where load() runs to completion before anything else can interleave.

Why aiobotocore triggers it easily

In aiobotocore, sharing one AioSession across many concurrent tasks is idiomatic and encouraged. The root cause is the same: _visited_profiles is an instance variable initialized once in __init__ and never reset in load():

# botocore AssumeRoleProvider.__init__ — initialized once
self._visited_profiles = [self._profile_name]  # e.g. ['a']

# AioAssumeRoleProvider.load() has no reset
async def load(self):
    self._loaded_config = self._load_config()
    ...

With async tasks:

Task 1                                    Task 2
─────────────────────────────────────────────────────────────
load() → _visited_profiles = ['a']
_resolve_source_credentials('b')
_visited_profiles.append('b') → ['a','b']
await <STS/IMDS call>  ← yields here

                                          load() called
                                          _validate_source_profile('b')
                                          'b' in _visited_profiles? → TRUE ❌

Why botocore itself doesn't reset _visited_profiles either

Interestingly, botocore's load() also does not reset _visited_profiles before resolving. This means:

• Calling load() twice on the same provider instance in botocore would also fail (the second call would see leftover entries from the first).
• This is presumably fine in botocore because CredentialResolver only calls each provider's load() once and caches the result.

In aiobotocore, multiple concurrent coroutines enter load() before the first completes, so the "only called once" assumption is violated.

Conclusion

This is effectively aiobotocore-specific in practice, because:

1. Concurrent async tasks sharing a session is natural and common in asyncio code
2. Synchronous botocore users would need to deliberately share a session across threads to hit this

The fix suggested in the previous comment (resetting _visited_profiles at the start of load()) is the right approach. It's also worth noting that botocore itself could benefit from the same defensive reset, but that's a separate upstream concern.

[thehesiod]

@claude can you create a PR with a proposed fix with test?

[thehesiod]

@jakob-keller eh? ;) first attempt, not too bad. I'm working on optimizing the tool calling, it made a ton of missteps on the way to the pr

[thehesiod]

lol from claude's own analysis:

  What happened (chronological)

  Phase 1 — Research (efficient, ~25s)

  1. gh issue view 1455 — read the issue
  2. gh issue view 1455 --comments — read comments
  3. grep _visited_profiles in credentials.py — found the code
  4. Read CLAUDE.md
  5. Several grep / python3 -c inspect.getsource(...) calls to understand botocore's base class

  Phase 2 — Write code (efficient, ~25s)

  6. Read test files, found test patterns
  7. Two Edit calls to modify credentials.py and add a test

  Phase 3 — Test environment disaster (~3 minutes wasted)