Prior to this change the behavior of the library has been

to refresh the credential from IAM only after its in the last
15 seconds of its life according to its expired_at field

This poses a problem when using the library to generate short
lived pre-signed URLs in AWS S3 where the issued URLs
with temporary tokens expire the same time as the IAM credential
that generated them:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html

For example, if the credentials is in its last 16 seconds of life
and a token has been generated for pre-signed URLs with a life
request of 10 minutes, the URL ends up living only 16 seconds.

By allowing the caller to configure the refresh threshold a
larger buffer of time can be specified when using IAM credentials
allowing for the generated URLs to life a more definitive time.

This change adds a configuration option that can be passed by
callers to expand (or narrow) the threshold of time used by the
refresh credentials mechanism to decide if a new token needs
to be fetched.

Configuration option is called
'aws_credentials_refresh_threshold_seconds'

Following the same example, setting this new option to 10 minutes
can help guarantee that pre-signed URLs generated for 10 minutes
will always last that long.

README.md

@@ -105,6 +105,48 @@ directory.files
 directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60)
 ```
 
+##### Controlling credential refresh time with IAM authentication
+
+When using IAM authentication with
+[temporary security credentials](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html),
+generated S3 pre-signed URLs
+[only last as long as the temporary credential](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html).
+
+Generating the URLs in the following manner will return a URL
+that will not last as long as its requested expiration time if
+the remainder of the authentication token lifetime was shorter.
+
+```ruby
+s3 = Fog::Storage.new(provider: 'AWS', use_iam_auth: true)
+directory = s3.directories.get('gaudi-portal-dev', prefix: 'user/1/')
+
+directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60)
+```
+
+By default the temporary credentials in use are refreshed only within the last
+15 seconds of its expiration time. The URL requested with 60 seconds lifetime
+using the above example will only remain valid for 15 seconds in the worst case.
+
+The problem can be avoided by refreshing the token early and often,
+by setting configuration `aws_credentials_refresh_threshold_seconds` (default: 15)
+which controls the time when the refresh must occur. It is expressed in seconds
+before the temporary credential's expiration time.
+
+The following example can ensure pre-signed URLs last as long as 60 seconds
+by automatically refreshing the credentials when its remainder lifetime
+is lower than 60 seconds:
+
+```ruby
+s3 = Fog::Storage.new(
+  provider: 'AWS',
+  use_iam_auth: true,
+  aws_credentials_refresh_threshold_seconds: 60
+)
+directory = s3.directories.get('gaudi-portal-dev', prefix: 'user/1/')
+
+directory.files.new(key: 'user/1/Gemfile').url(Time.now + 60)
+```
+
 #### Copying a file
 
 ```ruby

lib/fog/aws/credential_fetcher.rb

@@ -116,10 +116,17 @@ module Fog
 
         private
 
+        # When defined, 'aws_credentials_refresh_threshold_seconds' controls
+        # when the credential needs to be refreshed, expressed in seconds before
+        # the current credential's expiration time
+        def credentials_refresh_threshold
+          @aws_credentials_refresh_threshold_seconds || 15
+        end
+
         def credentials_expired?
           @use_iam_profile &&
             (!@aws_credentials_expire_at ||
-             (@aws_credentials_expire_at && Fog::Time.now > @aws_credentials_expire_at - 15)) #new credentials become available from around 5 minutes before expiration time
+             (@aws_credentials_expire_at && Fog::Time.now > @aws_credentials_expire_at - credentials_refresh_threshold)) #new credentials become available from around 5 minutes before expiration time
         end
 
         def refresh_credentials

lib/fog/aws/storage.rb

@@ -43,7 +43,7 @@ module Fog
       ]
 
       requires :aws_access_key_id, :aws_secret_access_key
-      recognizes :endpoint, :region, :host, :port, :scheme, :persistent, :use_iam_profile, :aws_session_token, :aws_credentials_expire_at, :path_style, :acceleration, :instrumentor, :instrumentor_name, :aws_signature_version, :enable_signature_v4_streaming, :virtual_host, :cname
+      recognizes :endpoint, :region, :host, :port, :scheme, :persistent, :use_iam_profile, :aws_session_token, :aws_credentials_expire_at, :path_style, :acceleration, :instrumentor, :instrumentor_name, :aws_signature_version, :enable_signature_v4_streaming, :virtual_host, :cname, :aws_credentials_refresh_threshold_seconds
 
       secrets    :aws_secret_access_key, :hmac
 
@@ -466,6 +466,8 @@ module Fog
         end
 
         def setup_credentials(options)
+          @aws_credentials_refresh_threshold_seconds = options[:aws_credentials_refresh_threshold_seconds]
+
           @aws_access_key_id = options[:aws_access_key_id]
           @aws_secret_access_key = options[:aws_secret_access_key]
           @aws_session_token     = options[:aws_session_token]
@@ -540,6 +542,8 @@ module Fog
 
 
         def setup_credentials(options)
+          @aws_credentials_refresh_threshold_seconds = options[:aws_credentials_refresh_threshold_seconds]
+
           @aws_access_key_id     = options[:aws_access_key_id]
           @aws_secret_access_key = options[:aws_secret_access_key]
           @aws_session_token     = options[:aws_session_token]

tests/credentials_tests.rb

@@ -101,25 +101,61 @@ Shindo.tests('AWS | credentials', ['aws']) do
 
     ENV['AWS_WEB_IDENTITY_TOKEN_FILE'] = nil
 
-    compute = Fog::AWS::Compute.new(use_iam_profile: true)
+    storage = Fog::Storage.new(
+      :provider => 'AWS',
+      :region => 'us-east-1',
+      :use_iam_profile => true,
+      :aws_credentials_refresh_threshold_seconds => 30)
 
-    tests('#refresh_credentials_if_expired') do
-      returns(nil) { compute.refresh_credentials_if_expired }
+    tests('#credentials_refresh_threshold') do
+      returns(30) { storage.send(:credentials_refresh_threshold) }
     end
 
-    credentials['AccessKeyId'] = 'newkey'
-    credentials['SecretAccessKey'] = 'newsecret'
+    Fog::Time.now = storage.instance_variable_get(:@aws_credentials_expire_at) - 31
+    tests('#refresh_credentials_if_expired before credentials have expired and before refresh threshold') do
+      returns(nil) { storage.refresh_credentials_if_expired }
+      returns('dummykey') { storage.instance_variable_get(:@aws_access_key_id) }
+      returns('dummysecret') { storage.instance_variable_get(:@aws_secret_access_key) }
+      returns(expires_at) { storage.instance_variable_get(:@aws_credentials_expire_at) }
+    end
+    Fog::Time.now = Time.now
+
+    credentials['AccessKeyId'] = 'newkey-1'
+    credentials['SecretAccessKey'] = 'newsecret-1'
     credentials['Expiration'] = (expires_at + 10).xmlschema
 
     Excon.stub({ method: :get, path: '/latest/meta-data/iam/security-credentials/arole' }, { status: 200, body: Fog::JSON.encode(credentials) })
 
-    Fog::Time.now = expires_at + 1
-    tests('#refresh_credentials_if_expired') do
-      returns(true) { compute.refresh_credentials_if_expired }
-      returns('newkey') { compute.instance_variable_get(:@aws_access_key_id) }
+    Fog::Time.now = storage.instance_variable_get(:@aws_credentials_expire_at) - 29
+    tests('#refresh_credentials_if_expired after refresh threshold is crossed but before expiration') do
+      returns(true) { storage.refresh_credentials_if_expired }
+      returns('newkey-1') { storage.instance_variable_get(:@aws_access_key_id) }
+      returns('newsecret-1') { storage.instance_variable_get(:@aws_secret_access_key) }
+      returns(expires_at + 10) { storage.instance_variable_get(:@aws_credentials_expire_at) }
     end
     Fog::Time.now = Time.now
 
+    credentials['AccessKeyId'] = 'newkey-2'
+    credentials['SecretAccessKey'] = 'newsecret-2'
+    credentials['Expiration'] = (expires_at + 20).xmlschema
+
+    Excon.stub({ method: :get, path: '/latest/meta-data/iam/security-credentials/arole' }, { status: 200, body: Fog::JSON.encode(credentials) })
+
+    Fog::Time.now = storage.instance_variable_get(:@aws_credentials_expire_at) + 1
+    tests('#refresh_credentials_if_expired after credentials have expired') do
+      returns(true) { storage.refresh_credentials_if_expired }
+      returns('newkey-2') { storage.instance_variable_get(:@aws_access_key_id) }
+      returns('newsecret-2') { storage.instance_variable_get(:@aws_secret_access_key) }
+      returns(expires_at + 20) { storage.instance_variable_get(:@aws_credentials_expire_at) }
+    end
+    Fog::Time.now = Time.now
+
+    compute = Fog::AWS::Compute.new(use_iam_profile: true)
+
+    tests('#credentials_refresh_threshold when "aws_credentials_refresh_threshold_seconds" is unspecified') do
+      returns(15) { compute.send(:credentials_refresh_threshold) }
+    end
+
     default_credentials = Fog::AWS::Compute.fetch_credentials({})
     tests('#fetch_credentials when the url 404s') do
       Excon.stub({ method: :put, path: '/latest/api/token' }, { status: 404, body: 'not found' })