Skip to content

PROVIDERS.md

Latest commit

 

History

History
607 lines (468 loc) · 21.1 KB

PROVIDERS.md

File metadata and controls

607 lines (468 loc) · 21.1 KB

Providers

Amazon Web Services (AWS)

Amazon Web Services can be integrated by using the following configuration block.

- # provider is the name of the provider
  provider: aws
  # id is the name defined by user for filtering (optional)
  id: staging
  # aws_access_key is the access key for AWS account
  aws_access_key: $AWS_ACCESS_KEY
  # aws_secret_key is the secret key for AWS account
  aws_secret_key: $AWS_SECRET_KEY
  # aws_session_token session token for temporary security credentials retrieved via STS (optional)
  aws_session_token: $AWS_SESSION_TOKEN
  # assume_role_arn is the arn of the role to assume (optional)
  assume_role_arn: $AWS_ASSUME_ROLE_ARN
  # external_id is the external id for the role to assume (required if assume_role_arn is provided)
  external_id: $AWS_EXTERNAL_ID
  # assume_role_session_name is the name of the session for the role to assume (required if assume_role_arn is provided)
  assume_role_session_name: $AWS_ASSUME_ROLE_SESSION_NAME
 # assume_role_name is the name of the role to assume (optional)
 assume_role_name: $AWS_ASSUME_ROLE_NAME
 # account_ids is the aws account ids which has similar assumed role name (optional)
 account_ids:
   - $AWS_ACCOUNT_ID_1
   - $AWS_ACCOUNT_ID_2

aws_access_key and aws_secret_key can be generated in the IAM console. We recommend creating a new IAM user with Read Only permissions and providing the access token for the user.

Scopes Required -

  1. EC2
  2. Route53

References -

  1. https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_iam_read-only-console.html
  2. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
  3. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html

Google Cloud Platform (GCP)

Google Cloud Platform supports two discovery approaches and two authentication modes:

Authentication Modes

A. Traditional Authentication (Static Credentials)

  • Service account keys (JSON files) - long-lived
  • Application Default Credentials (ADC) - often long-lived

B. Short-lived Credentials (Recommended for Enhanced Security)

  • Generate temporary access tokens (up to 1 hour)
  • Eliminates reliance on static service account keys
  • Uses Service Account Credentials API for token generation
  • Supports service account impersonation

1. Individual Service APIs (Project-Level Discovery)

Option 1: Traditional Static Credentials

- # provider is the name of the provider
  provider: gcp
  # id is the name defined by user for filtering (optional)
  id: project-discovery
  # gcp_service_account_key is the key token of service account.
  gcp_service_account_key: '{
    "type": "service_account",
    "project_id": "your-project-id",
    "private_key_id": "...",
    "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
    "client_email": "cloudlist-sa@your-project-id.iam.gserviceaccount.com",
    "client_id": "...",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/cloudlist-sa%40your-project-id.iam.gserviceaccount.com",
    "universe_domain": "googleapis.com"
  }'

Option 2: Short-lived Credentials (Developer Workflow - Zero Keys)

- provider: gcp
  id: dev-discovery
  use_short_lived_credentials: true
  service_account_email: "cloudlist@project.iam.gserviceaccount.com"
  # Uses: gcloud auth login → ADC → short-lived token
  # No service account key file needed!

Option 3: Short-lived Credentials (CI/CD with Minimal Permissions)

- provider: gcp
  id: ci-discovery
  use_short_lived_credentials: true
  service_account_email: "powerful-sa@project.iam.gserviceaccount.com"
  source_credentials: "minimal-ci-sa.json"  # Only has impersonation permission
  token_lifetime: "7200s"  # 2 hours

Option 4: Short-lived Credentials (GKE/Compute Engine - Zero Secrets)

- provider: gcp
  id: workload-discovery
  use_short_lived_credentials: true
  service_account_email: "cloudlist@project.iam.gserviceaccount.com"
  # Uses workload identity automatically

Option 5: Short-lived Credentials (Migration from Existing Keys)

- provider: gcp
  id: migrating-discovery
  use_short_lived_credentials: true
  service_account_email: "cloudlist@project.iam.gserviceaccount.com"
  gcp_service_account_key: "existing-key.json"  # Generates short-lived from static key
  token_lifetime: "3600s"  # 1 hour (default)

Required Scopes (Traditional Authentication):

  1. roles/compute.viewer - Compute instances and forwarding rules
  2. roles/dns.reader - DNS records
  3. roles/storage.objectViewer - Storage buckets
  4. roles/run.viewer - Cloud Run services
  5. roles/cloudfunctions.viewer - Cloud Functions
  6. roles/container.viewer - GKE clusters
  7. roles/tpu.viewer - TPU nodes
  8. roles/file.viewer - Filestore instances
  9. roles/resourcemanager.viewer - List projects

Additional Requirements for Short-lived Credentials:

  • Source credentials need: roles/iam.serviceAccountTokenCreator or iam.serviceAccounts.generateAccessToken permission on the target service account
  • Target service account needs: Same viewer roles listed above
  • Service Account Credentials API must be enabled in the project

Configuration Parameters:

  • use_short_lived_credentials (bool): Enable short-lived token generation (default: false)
  • service_account_email (string, required if short-lived): Target service account to impersonate
  • source_credentials (string, optional): Path to source credentials file (uses ADC if not provided)
  • token_lifetime (string, optional): Token lifetime in seconds (e.g., "3600s") or Go duration format (e.g., "1h"). Range: 1s to 3600s (1 hour). Default: "3600s"
  • project_ids (list, optional): Comma-separated/list of project IDs to enumerate. When provided, Cloudlist skips discovery in every other accessible project, both for individual APIs and the organization-level Asset API.

2. Organization-Level Asset API (Organization-Wide Discovery)

Traditional Authentication:

- # provider is the name of the provider
  provider: gcp
  # id is the name defined by user for filtering (optional)
  id: org-discovery
  # organization_id enables Asset API for organization-wide discovery
  organization_id: "123456789012"
  # gcp_service_account_key with organization-level permissions
  gcp_service_account_key: '{
    "type": "service_account",
    "project_id": "your-project-id",
    "private_key_id": "...",
    "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
    "client_email": "asset-viewer-sa@your-project-id.iam.gserviceaccount.com",
    "client_id": "...",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/asset-viewer-sa%40your-project-id.iam.gserviceaccount.com",
    "universe_domain": "googleapis.com"
  }'

Short-lived Credentials (Organization-Level):

- provider: gcp
  id: org-discovery-secure
  organization_id: "123456789012"
  use_short_lived_credentials: true
  service_account_email: "asset-viewer-sa@project.iam.gserviceaccount.com"
  token_lifetime: "7200s"  # 2 hours
  project_ids:
    - security-core
    - shared-infra

Add project_ids to either configuration style to limit enumeration strictly to the listed projects (Cloud Asset API requests are filtered too), which is helpful for large organizations or delegated-access service accounts.

Required Organization-Level Roles:

  1. roles/cloudasset.viewer - Core Asset API access
  2. roles/resourcemanager.viewer - List projects in organization
  3. (For short-lived) roles/iam.serviceAccountTokenCreator - On source credentials

Key Differences:

  • Individual APIs: Fast, project-specific, detailed results
  • Asset API: Comprehensive, organization-wide, higher resource count
  • Short-lived Credentials: Enhanced security, tokens auto-expire (up to 1 hour)

📚 For detailed setup instructions, see: docs/GCP_ASSET_API.md

References -

  1. https://cloud.google.com/asset-inventory/docs/overview
  2. https://cloud.google.com/iam/docs/creating-managing-service-accounts
  3. https://cloud.google.com/iam/docs/understanding-roles
  4. https://cloud.google.com/iam/docs/service-account-creds (Short-lived credentials)
  5. https://cloud.google.com/docs/authentication/provide-credentials-adc (Application Default Credentials)

Microsoft Azure

Microsoft Azure can be integrated by using the following configuration block.

- # provider is the name of the provider
 provider: azure
 # id is the name defined by user for filtering (optional)
 id: staging
 # client_id is the client ID of registered application of the azure account (not requuired if using cli auth)
 client_id: $AZURE_CLIENT_ID
 # client_secret is the secret ID of registered application of the zure account (not requuired if using cli auth)
 client_secret: $AZURE_CLIENT_SECRET
 # tenant_id is the tenant ID of registered application of the azure account (not requuired if using cli auth)
 tenant_id: $AZURE_TENANT_ID
 #subscription_id is the azure subscription id
 subscription_id: $AZURE_SUBSCRIPTION_ID
 #use_cli_auth if set to true cloudlist will use azure cli auth
 use_cli_auth: true

tenant_id, client_id, client_secret can be obtained/generated from All services > Azure Active Directory > App registrations subscription_id can be retrieved from All services > Subscriptions

To use cli auth set use_cli_auth value to true and run az login in the terminal

References -

  1. https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli
  2. https://docs.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest#az_ad_sp_create_for_rbac
  3. https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli

DigitalOcean (DO)

Digitalocean can be integrated by using the following configuration block.

- # provider is the name of the provider
  provider: do
  # digitalocean_token is the API key for digitalocean cloud platform
  digitalocean_token: $DIGITALOCEAN_TOKEN

digitalocean_token can be generated from the Digitalocean Control Panel. We recommend only giving Read Access to the token.

References -

  1. https://www.digitalocean.com/docs/apis-clis/api/create-personal-access-token/

Scaleway (SCW)

Scaleway can be integrated by using the following configuration block.

- # provider is the name of the provider
  provider: scw
  # scaleway_access_key is the access key for scaleway API
  scaleway_access_key: $SCALEWAY_ACCESS_KEY
  # scaleway_access_token is the access token for scaleway API
  scaleway_access_token: $SCALEWAY_ACCESS_TOKEN

scaleway_access_key and scaleway_access_token can be generated from the Credentials Options in scaleway console.

References -

  1. https://www.scaleway.com/en/docs/generate-api-keys/

ArvanCloud

ArvanCloud can be integrated by using the following configuration block.

- # provider is the name of the provider
  provider: arvancloud # or r1c
  # api_key is the api_key for arvancloud
  api_key: $R1C_API_KEY

The api_key can be generated from ArvanCloud Machine User manager.

References -

  1. https://www.arvancloud.ir/en/dev/api
  2. https://docs.arvancloud.ir/en/developer-tools/api/api-usage
  3. https://panel.arvancloud.ir/profile/machine-user

Cloudflare

Cloudflare can be integrated by using the following configuration block.

- # provider is the name of the provider
  provider: cloudflare
  # email is the email for cloudflare
  email: $CF_EMAIL
  # api_key is the api_key for cloudflare
  api_key: $CF_API_KEY
  # api_token is the scoped_api_token for cloudflare (optional)
  api_token: $CF_API_TOKEN

api_key can be generated from Cloudflare API Key manager. It needs to be Global API Key due to limitation of cloudflare new API tokens.

References -

  1. https://developers.cloudflare.com/api/keys

Heroku

Heroku can be integrated by using the following configuration block.

- # provider is the name of the provider
 provider: heroku
 # id is the name defined by user for filtering (optional)
 id: staging
 # heroku_api_token is the authorization token for Heroku account
 heroku_api_token: $HEROKU_API_TOKEN

heroku_api_token can be generated from https://dashboard.heroku.com/account/applications/authorizations/new

It can also be created with the Heroku CLI by running:

$ heroku authorizations:create -d "brief description of token"
Creating OAuth Authorization... done
Client:      <none>
ID:          a6e98151-f242-4592-b107-25fbac5ab410
Description: brief description of token
Scope:       global
Token:       cf0e05d9-4eca-4948-a012-b9xxxxxxxxxx
Updated at:  Fri Jun 16 2021 13:26:56 GMT-0700 (PDT) (less than a minute ago)

References -

  1. https://devcenter.heroku.com/articles/platform-api-quickstart#authentication

Fastly

Fastly can be integrated by using the following configuration block.

- # provider is the name of the provider
 provider: fastly
 # id is the name defined by user for filtering (optional)
 id: staging
 # fastly_api_key is the personal API token for fastly account
 fastly_api_key: $FASTLY_API_KEY

fastly_api_key can be generated from https://manage.fastly.com/account/personal/tokens

References -

  1. https://docs.fastly.com/en/guides/using-api-tokens#creating-api-tokens

Linode

Linode can be integrated by using the following configuration block.

- # provider is the name of the provider
 provider: linode
 # id is the name defined by user for filtering (optional)
 id: staging
 # linode_personal_access_token is the personal access token for Linode account
 linode_personal_access_token: $LINODE_PERSONAL_ACCESS_TOKEN

linode_personal_access_token can be created from https://cloud.linode.com/id/tokens. Minimum scope needed is Read Only for Linodes resource.

References -

  1. https://www.linode.com/docs/guides/getting-started-with-the-linode-api/#get-an-access-token

Namecheap

Namecheap can be integrated by using the following configuration block.

- # provider is the name of the provider
  provider: namecheap
  # id is the name defined by user for filtering (optional)
  id: staging
  # namecheap_api_key is the api key for namecheap account
  namecheap_api_key: $NAMECHEAP_API_KEY
  # namecheap_user_name is the username of the namecheap account
  namecheap_user_name: $NAMECHEAP_USER_NAME

Namecheap API Access can be enabled by visiting https://ap.www.namecheap.com/settings/tools/apiaccess/ and then:

  • Toggle ON API Access switch
  • Add your public IP to Whitelistted IPs

References -

Alibaba Cloud

Alibaba Cloud can be integrated by using the following configuration block.

- # provider is the name of the provider
 provider: alibaba
 # id is the name defined by user for filtering (optional)
 id: staging
 # alibaba_region_id is the region id of the resources
 alibaba_region_id: $ALIBABA_REGION_ID
 # alibaba_access_key is the access key ID for alibaba cloud account
 alibaba_access_key: $ALIBABA_ACCESS_KEY
 # alibaba_access_key_secret is the secret access key for alibaba cloud account
 alibaba_access_key_secret: $ALIBABA_ACCESS_KEY_SECRET

Alibaba Cloud Access Key ID and Secret can be created by visiting https://ram.console.aliyun.com/manage/ak

References -

Terraform

Terraform can be integrated by using the following configuration block.

- # provider is the name of the provider
 provider: terraform
 # id is the name defined by user for filtering (optional)
 id: staging
 #tf_state_file is the location of terraform state file (terraform.tfsate) 
 tf_state_file: path/to/terraform.tfstate

Hashicorp Consul

Hashicorp Consul can be integrated by using the following configuration block.

- # provider is the name of the provider
  provider: consul
  # consul_url is the url for consul server
  consul_url: http://localhost:8500/
  # id is the name defined by user for filtering (optional)
  id: staging
  # consul_ca_file is the path to consul CA file
  # consul_ca_file: <path-to-ca-file>.pem
  # consul_cert_file is the path to consul Certificate file
  # consul_cert_file: <path-to-cert-file>.pem
  # consul_key_file is the path to consul Certificate Key file
  # consul_key_file: <path-to-key-file>.pem
  # consul_http_token is the consul authentication token
  # consul_http_token: <consul-token>
  # consul_http_auth is the consul http auth value
  # consul_http_auth: <consul-http-auth-value>

Specifying https in the consul_url automatically turns SSL to on. All the fields are optional except the consul_url.

References -

Hashicorp Nomad

Hashicorp Nomad can be integrated by using the following configuration block.

- # provider is the name of the provider
  provider: nomad
  # nomad_url is the url for nomad server
  nomad_url: http://127.0.0.1:4646/
  # id is the name defined by user for filtering (optional)
  id: staging
  # nomad_ca_file is the path to nomad CA file
  # nomad_ca_file: <path-to-ca-file>.pem
  # nomad_cert_file is the path to nomad Certificate file
  # nomad_cert_file: <path-to-cert-file>.pem
  # nomad_key_file is the path to nomad Certificate Key file
  # nomad_key_file: <path-to-key-file>.pem
  # nomad_token is the nomad authentication token
  # nomad_token: <nomad-token>
  # nomad_http_auth is the nomad http auth value
  # nomad_http_auth: <nomad-http-auth-value>

Specifying https in the nomad_url automatically turns SSL to on. All the fields are optional except the nomad_url.

References -

Hetzner Cloud

Hetzner Cloud can be integrated by using the following configuration block.

- # provider is the name of the provider
 provider: hetzner
 # id is the name defined by user for filtering (optional)
 id: staging
 # auth_token is the is the hetzner authentication token
 auth_token: $HETZNER_AUTH_TOKEN

References -

Kubernetes

To integrate Kubernetes, utilize the configuration block outlined below. This block allows you to specify Kubernetes connection details either through a file path or directly by providing the encoded kubeconfig content. If both kubeconfig_file and kubeconfig_encoded are specified, the kubeconfig_encoded will take precedence.

- # This identifies the provider in use
  provider: kubernetes
  # User-defined identifier for filtering (optional).
  id: staging
  # Path to the kubeconfig file.
  kubeconfig_file: path/to/kubeconfig
  # Base64 encoded kubeconfig, $ cat $KUBECONFIG | base64
  kubeconfig_encoded: <encoded-kubeconfig>
  # The context to use from the kubeconfig (optional). If omitted, the default is the current-context as defined in the kubeconfig.
  context: <context-name>

References -

DNSSimple

DNSSimple can be integrated by using the following configuration block.

- # provider is the name of the provider
  provider: dnssimple
  # id is the name defined by user for filtering (optional)
  id: main
  # dnssimple_api_token is the API token for DNSSimple
  dnssimple_api_token: $DNSSIMPLE_API_TOKEN

The dnssimple_api_token can be generated from the DNSSimple account settings under "API Access". You will need to create an API v2 token with appropriate permissions to access your domains and DNS records.

References -

  1. https://developer.dnsimple.com/v2/
  2. https://support.dnsimple.com/articles/api-access-token/

OVH

Cloudlist supports fetching DNS records from OVH.

  • Provider key: ovh
  • Services: dns
  • id: An arbitrary label you choose to tag resources. It helps distinguish multiple OVH accounts/configs.
  • Required auth:
    • application_key
    • application_secret
    • consumer_key
  • Endpoint: OVH API endpoint. Defaults to ovh-eu if omitted. Common values: ovh-eu, ovh-ca, ovh-us.

Configuration example (from ovh.yaml):

- provider: ovh
  id: ovh-prod
  endpoint: ovh-ca
  application_key: $OVH_APP_KEY
  application_secret: $OVH_APP_SECRET
  consumer_key: $OVH_CONSUMER_KEY

References -

  1. https://eu.api.ovh.com/console/?section=%2Fdomain&branch=v1
  2. https://api.ovh.com/createToken/
  3. https://help.ovhcloud.com/csm/en-gb-api-getting-started-ovhcloud-api