Credential handling updates (#20)

* add a seperate option for premium accounts to use consumer_key and secret values and inherently generate the bearer token on startup

* b64 encode expects a byte string

* after some more testing, better understood the required headers

* updated docsstrings around credentials allowing consumer key/secrets

* updated docs for credentials

* simplifying bearer token retrieval

* updated base readme with new docs

* update version

Author: Aaron Gonzales

README.rst

@@ -49,16 +49,23 @@ Credential Handling
 
 The premium and enterprise Search APIs use different authentication
 methods and we attempt to provide a seamless way to handle
-authentication for all customers.
+authentication for all customers. We know credentials can be tricking or
+annoying - please read this in its entirety.
 
 Premium clients will require the ``bearer_token`` and ``endpoint``
 fields; Enterprise clients require ``username``, ``password``, and
 ``endpoint``. If you do not specify the ``account_type``, we attempt to
 discern the account type and declare a warning about this behavior.
 
 For premium search products, we are using app-only authentication and
-the bearer tokens are not delivered with an expiration time. They can be
-invalidated. Please see
+the bearer tokens are not delivered with an expiration time. You can
+provide either: - your application key and secret (the library will
+handle bearer-token authentication) - a bearer token that you get
+yourself
+
+Many developers might find providing your application key and secret
+more straightforward and letting this library manage your bearer token
+generation for you. Please see
 `here <https://developer.twitter.com/en/docs/basics/authentication/overview/application-only>`__
 for an overview of the premium authentication method.
 
@@ -77,7 +84,8 @@ this:
     search_tweets_api:
       account_type: premium
       endpoint: <FULL_URL_OF_ENDPOINT>
-      bearer_token: <TOKEN>
+      consumer_key: <CONSUMER_KEY>
+      consumer_secret: <CONSUMER_SECRET>
 
 For enterprise customers, the simplest credential file should look like
 this:
@@ -109,7 +117,10 @@ line app. An example:
     search_tweets_30_day_dev:
       account_type: premium
       endpoint: <FULL_URL_OF_ENDPOINT>
-      bearer_token: <TOKEN>
+      consumer_key: <KEY>
+      consumer_secret: <SECRET>
+      (optional) bearer_token: <TOKEN>
+
 
     search_tweets_30_day_prod:
       account_type: premium
@@ -139,6 +150,8 @@ can set the appropriate variables for your product of the following:
     export SEARCHTWEETS_PASSWORD=
     export SEARCHTWEETS_BEARER_TOKEN=
     export SEARCHTWEETS_ACCOUNT_TYPE=
+    export SEARCHTWEETS_CONSUMER_KEY=
+    export SEARCHTWEETS_CONSUMER_SECRET=
 
 The ``load_credentials`` function will attempt to find these variables
 if it cannot load fields from the YAML file, and it will **overwrite any

docs/source/conf.py

@@ -66,9 +66,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '1.2'
+version = '1.3'
 # The full version, including alpha/beta/rc tags.
-release = '1.2.1'
+release = '1.3.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

examples/credential_handling.ipynb

@@ -6,12 +6,17 @@
    "source": [
     "# Credential Handling\n",
     "\n",
-    "The premium and enterprise Search APIs use different authentication methods and we attempt to provide a seamless way to handle authentication for all customers. \n",
+    "The premium and enterprise Search APIs use different authentication methods and we attempt to provide a seamless way to handle authentication for all customers. We know credentials can be tricking or annoying - please read this in its entirety.\n",
+    "\n",
     "\n",
     "Premium clients will require the `bearer_token` and `endpoint` fields; Enterprise clients require `username`, `password`, and `endpoint`.\n",
     "If you do not specify the `account_type`, we attempt to discern the account type and declare a warning about this behavior.\n",
     "\n",
-    "For premium search products, we are using app-only authentication and the bearer tokens are not delivered with an expiration time. They can be invalidated. Please see [here](https://developer.twitter.com/en/docs/basics/authentication/overview/application-only) for an overview of the premium authentication method.\n",
+    "For premium search products, we are using app-only authentication and the bearer tokens are not delivered with an expiration time. You can provide either:\n",
+    "- your application key and secret (the library will handle bearer-token authentication)\n",
+    "- a bearer token that you get yourself\n",
+    "\n",
+    "Many developers might find providing your application key and secret more straightforward and letting this library manage your bearer token generation for you. Please see [here](https://developer.twitter.com/en/docs/basics/authentication/overview/application-only) for an overview of the premium authentication method.\n",
     "\n",
     "We support both YAML-file based methods and environment variables for storing credentials, and provide flexible handling with sensible defaults.\n",
     "\n",
@@ -25,7 +30,8 @@
     "search_tweets_api:\n",
     "  account_type: premium\n",
     "  endpoint: <FULL_URL_OF_ENDPOINT>\n",
-    "  bearer_token: <TOKEN>\n",
+    "  consumer_key: <CONSUMER_KEY>\n",
+    "  consumer_secret: <CONSUMER_SECRET>\n",
     "```\n",
     "\n",
     "For enterprise customers, the simplest credential file should look like this:\n",
@@ -37,14 +43,18 @@
     "  endpoint: <FULL_URL_OF_ENDPOINT>\n",
     "  username: <USERNAME>\n",
     "  password: <PW>\n",
-    "```\n",
-    "\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
     "By default, this library expects this file at `\"~/.twitter_keys.yaml\"`, but you can pass the relevant location as needed, either with the ``--credential-file`` flag for the command-line app or as demonstrated below in a Python program.\n",
     "\n",
     "Both above examples require no special command-line arguments or in-program arguments.  The credential parsing methods, unless otherwise specified, will look for a YAML key called `search_tweets_api`.\n",
     "\n",
     "\n",
-    "\n",
     "For developers who have multiple endpoints and/or search products, you can keep all credentials in the same file and specify specific keys to use. `--credential-file-key` specifies this behavior in the command line app. An example:\n",
     "\n",
     "\n",
@@ -53,7 +63,10 @@
     "search_tweets_30_day_dev:\n",
     "  account_type: premium\n",
     "  endpoint: <FULL_URL_OF_ENDPOINT>\n",
-    "  bearer_token: <TOKEN>\n",
+    "  consumer_key: <KEY>\n",
+    "  consumer_secret: <SECRET>\n",
+    "  (optional) bearer_token: <TOKEN>\n",
+    "  \n",
     "  \n",
     "search_tweets_30_day_prod:\n",
     "  account_type: premium\n",
@@ -87,6 +100,8 @@
     "export SEARCHTWEETS_PASSWORD=\n",
     "export SEARCHTWEETS_BEARER_TOKEN=\n",
     "export SEARCHTWEETS_ACCOUNT_TYPE=\n",
+    "export SEARCHTWEETS_CONSUMER_KEY=\n",
+    "export SEARCHTWEETS_CONSUMER_SECRET=\n",
     "```\n",
     "\n",
     "The `load_credentials` function will attempt to find these variables if it cannot load fields from the YAML file, and it will **overwrite any credentials from the YAML file that are present as environment variables** if they have been parsed. This behavior can be changed by setting the `load_credentials` parameter `env_overwrite` to `False`.\n",
@@ -97,7 +112,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 10,
    "metadata": {
     "collapsed": true
    },
@@ -108,7 +123,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 14,
    "metadata": {},
    "outputs": [
     {
@@ -119,7 +134,7 @@
        " 'username': '<MY_USERNAME>'}"
       ]
      },
-     "execution_count": 2,
+     "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -132,7 +147,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 15,
    "metadata": {},
    "outputs": [
     {
@@ -142,7 +157,7 @@
        " 'endpoint': 'https://api.twitter.com/1.1/tweets/search/30day/dev.json'}"
       ]
      },
-     "execution_count": 3,
+     "execution_count": 15,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -164,7 +179,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 16,
    "metadata": {},
    "outputs": [
     {
@@ -183,7 +198,7 @@
        " 'username': '<ENV_USERNAME>'}"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 16,
      "metadata": {},
      "output_type": "execute_result"
     }

examples/credential_handling.rst

@@ -4,16 +4,23 @@ Credential Handling
 
 The premium and enterprise Search APIs use different authentication
 methods and we attempt to provide a seamless way to handle
-authentication for all customers.
+authentication for all customers. We know credentials can be tricking or
+annoying - please read this in its entirety.
 
 Premium clients will require the ``bearer_token`` and ``endpoint``
 fields; Enterprise clients require ``username``, ``password``, and
 ``endpoint``. If you do not specify the ``account_type``, we attempt to
 discern the account type and declare a warning about this behavior.
 
 For premium search products, we are using app-only authentication and
-the bearer tokens are not delivered with an expiration time. They can be
-invalidated. Please see
+the bearer tokens are not delivered with an expiration time. You can
+provide either: - your application key and secret (the library will
+handle bearer-token authentication) - a bearer token that you get
+yourself
+
+Many developers might find providing your application key and secret
+more straightforward and letting this library manage your bearer token
+generation for you. Please see
 `here <https://developer.twitter.com/en/docs/basics/authentication/overview/application-only>`__
 for an overview of the premium authentication method.
 
@@ -33,7 +40,8 @@ this:
     search_tweets_api:
       account_type: premium
       endpoint: <FULL_URL_OF_ENDPOINT>
-      bearer_token: <TOKEN>
+      consumer_key: <CONSUMER_KEY>
+      consumer_secret: <CONSUMER_SECRET>
 
 For enterprise customers, the simplest credential file should look like
 this:
@@ -67,7 +75,10 @@ line app. An example:
     search_tweets_30_day_dev:
       account_type: premium
       endpoint: <FULL_URL_OF_ENDPOINT>
-      bearer_token: <TOKEN>
+      consumer_key: <KEY>
+      consumer_secret: <SECRET>
+      (optional) bearer_token: <TOKEN>
+      
       
     search_tweets_30_day_prod:
       account_type: premium
@@ -98,6 +109,8 @@ can set the appropriate variables for your product of the following:
     export SEARCHTWEETS_PASSWORD=
     export SEARCHTWEETS_BEARER_TOKEN=
     export SEARCHTWEETS_ACCOUNT_TYPE=
+    export SEARCHTWEETS_CONSUMER_KEY=
+    export SEARCHTWEETS_CONSUMER_SECRET=
 
 The ``load_credentials`` function will attempt to find these variables
 if it cannot load fields from the YAML file, and it will **overwrite any

searchtweets/credentials.py

@@ -11,8 +11,12 @@
 import os
 import logging
 import yaml
+import requests
+import base64
 from .utils import merge_dicts
 
+OAUTH_ENDPOINT = 'https://api.twitter.com/oauth2/token'
+
 __all__ = ["load_credentials"]
 
 logger = logging.getLogger(__name__)
@@ -45,7 +49,10 @@ def _load_env_credentials():
              "SEARCHTWEETS_USERNAME",
              "SEARCHTWEETS_PASSWORD",
              "SEARCHTWEETS_BEARER_TOKEN",
-             "SEARCHTWEETS_ACCOUNT_TYPE"]
+             "SEARCHTWEETS_ACCOUNT_TYPE",
+             "SEARCHTWEETS_CONSUMER_KEY",
+             "SEARCHTWEETS_CONSUMER_SECRET"
+             ]
     renamed = [var.replace("SEARCHTWEETS_", '').lower() for var in vars_]
 
     parsed = {r: os.environ.get(var) for (var, r) in zip(vars_, renamed)}
@@ -76,8 +83,16 @@ def _parse_credentials(search_creds, account_type):
 
     try:
         if account_type == "premium":
-            search_args = {"bearer_token": search_creds["bearer_token"],
-                           "endpoint": search_creds["endpoint"]}
+            if "bearer_token" not in search_creds:
+                if "consumer_key" in search_creds \
+                  and "consumer_secret" in search_creds:
+                    search_creds["bearer_token"] = _generate_bearer_token(
+                        search_creds["consumer_key"],
+                        search_creds["consumer_secret"])
+
+            search_args = {
+                "bearer_token": search_creds["bearer_token"],
+                "endpoint": search_creds["endpoint"]}
         if account_type == "enterprise":
             search_args = {"username": search_creds["username"],
                            "password": search_creds["password"],
@@ -96,35 +111,21 @@ def load_credentials(filename=None, account_type=None,
     """
     Handles credential management. Supports both YAML files and environment
     variables. A YAML file is preferred for simplicity and configureability.
-    A YAML credential file should look like this:
+    A YAML credential file should look something like this:
 
     .. code:: yaml
 
         <KEY>:
           endpoint: <FULL_URL_OF_ENDPOINT>
           username: <USERNAME>
           password: <PW>
+          consumer_key: <KEY>
+          consumer_secret: <SECRET>
           bearer_token: <TOKEN>
           account_type: <enterprise OR premium>
 
     with the appropriate fields filled out for your account. The top-level key
-    defaults to ``search_tweets_api`` but can be flexible, e.g.:
-
-    .. code:: yaml
-
-        premium_dev:
-          account_type: premium
-          endpoint: <FULL_URL_OF_ENDPOINT>
-          bearer_token: <TOKEN>
-
-        enterprise_dev:
-          account_type: enterprise
-          endpoint: <FULL_URL_OF_ENDPOINT>
-          username: <MY_USERNAME>
-          password: <MY_PASSWORD>
-
-    as this method supports a flexible interface for reading the
-    credential files. You can keep all of your credentials in the same file.
+    defaults to ``search_tweets_api`` but can be flexible.
 
     If a YAML file is not found or is missing keys, this function will check
     for this information in the environment variables that correspond to
@@ -136,9 +137,10 @@ def load_credentials(filename=None, account_type=None,
         SEARCHTWEETS_PASSWORD
         SEARCHTWEETS_BEARER_TOKEN
         SEARCHTWEETS_ACCOUNT_TYPE
+        ...
 
     Again, set the variables that correspond to your account information and
-    type.
+    type. See the main documentation for details and more examples.
 
 
     Args:
@@ -162,12 +164,12 @@ def load_credentials(filename=None, account_type=None,
         dict_keys(['bearer_token', 'endpoint'])
         >>> import os
         >>> os.environ["SEARCHTWEETS_ENDPOINT"] = "https://endpoint"
-        >>> os.environ["SEARCHTWEETS_USERNAME"] = "azaleszzz"
+        >>> os.environ["SEARCHTWEETS_USERNAME"] = "areallybadpassword"
         >>> os.environ["SEARCHTWEETS_PASSWORD"] = "<PW>"
         >>> load_credentials()
         {'endpoint': 'https://endpoint',
          'password': '<PW>',
-         'username': 'azaleszzz'}
+         'username': 'areallybadpassword'}
 
     """
     yaml_key = yaml_key if yaml_key is not None else "search_tweets_api"
@@ -183,3 +185,19 @@ def load_credentials(filename=None, account_type=None,
                    else merge_dicts(env_vars, yaml_vars))
     parsed_vars = _parse_credentials(merged_vars, account_type=account_type)
     return parsed_vars
+
+
+def _generate_bearer_token(consumer_key, consumer_secret):
+    """
+    Return the bearer token for a given pair of consumer key and secret values.
+    """
+    data = [('grant_type', 'client_credentials')]
+    resp = requests.post(OAUTH_ENDPOINT,
+                         data=data,
+                         auth=(consumer_key, consumer_secret))
+    logger.warning("Grabbing bearer token from OAUTH")
+    if resp.status_code >= 400:
+        logger.error(resp.text)
+        resp.raise_for_status()
+
+    return resp.json()['access_token']