[aws|cdn] AWS CDN models

This patch brings models for AWS CDN, including tests and documentation.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>

docs/cdn/index.markdown

@@ -46,12 +46,63 @@ Now you'll need to create a 'distribution' which represents a mapping from the C
       cdn.get_distribution(distribution_id).body['Status'] == 'Deployed'
     }
 
+Fog also supports models for the AWS CDN. The above code can also be written like this:
+
+    distribution = cdn.distributions.create( :custom_origin => {
+          'DNSName'               => 'www.example.com',
+          'OriginProtocolPolicy'  => 'match-viewer'
+        }, :enabled => true
+    })
+    
+    distribution.wait_for { ready? }
+
+Like other collections supported by Fog, it is also possible to browse the list of distributions:
+
+    cdn.distributions.all
+    
+Or get access to a distinct distribution by its identity:
+
+    cdn.distributions.get(distribution_id)
+
+
 ## Getting Served
 
 With the domain name from the distribution in hand you should now be ready to serve content from the edge.  All you need to do is start replacing urls like `http://www.example.com/stylesheets/foo.css` with `#{cdn_domain_name}/stylesheets/foo.css`. Just because you can do something doesn't always mean you should though.  Dynamic pages are not really well suited to CDN storage, since CDN content will be the same for every user.  Fortunately some of your most used content is a great fit.  By just switching over your images, javascripts and stylesheets you can have an impact for each and every one of your users.
 
 Congrats, your site is faster! By default the urls aren't very pretty, something like `http://d1xdx2sah5udd0.cloudfront.net/stylesheets/foo.css`.  Thankfully you can use CNAME config options to utilize something like `http://assets.example.com/stylesheets/foo.css`, if you are interested in learning more about this let me know in the comments.
 
+## Invalidating the CDN caches
+
+Sometimes, some part of the CDN cache needs to be invalidated because the origin changed and we need a faster propagation than waiting for the objects to expire by themselves. To do this, CloudFront supports creating <a href="http://docs.amazonwebservices.com/AmazonCloudFront/latest/APIReference/Actions_Invalidations.html">distributions invalidation</a>.
+
+An invalidation can be created with the following code:
+
+    # let's invalidate /test.html and /path/to/file.html
+    data = cdn.post_invalidation(distribution_id, [ "/test.html", "/path/to/file.html" ])
+    invalidation_id = data.body['Id']
+
+    Fog.wait_for { cdn.get_invalidation(distribution_id, invalidation_id).body['Status'] == 'Completed' }
+
+It is also possible to list past and current invalidation for a given distribution:
+
+    cdn.get_invalidation_list(distribution_id)
+
+The same can be written with Fog CDN model abstraction:
+
+    distribution = cdn.distributions.get(distribution_id)
+    
+    invalidation = distribution.invalidations.create(:paths => [ "/test.html", "/path/to/file.html" ])
+    invalidation.wait_for { ready? }
+    
+Listing invalidations is as simple as:
+
+    distribution.invalidations.all
+
+    # this returns only summarized invalidation
+    # to get access to the invalidations path:
+    distribution.invalidations.get(invalidation_id)
+
+
 ## Cleaning Up
 
 But, just in case you need to update things I'll run through how you can make changes. In my case I just want to clean up after myself, so I'll use the distribution_id and ETag from before to disable the distribution. We need to use the ETag as well because it provides a way to refer to different versions of the same distribution and ensures we are updating the version that we think we are.
@@ -79,4 +130,19 @@ Now you just need to wait for the update to happen like before and once its disa
     }
     cdn.delete_distribution(distribution_id, etag)
 
+This can also be written with CDN models as:
+
+    distribution = cdn.distributions.get(distribution_id)
+    
+    # make sure the distribution is deployed otherwise it can't be disabled
+    distribution.wait_for { ready? }
+    
+    distribution.disable
+    
+    # Disabling a distribution is a lengthy operation
+    distribution.wait_for { ready? }
+    
+    # and finally let's get rid of it
+    distribution.destroy
+
 Thats it, now go forth and speed up some load times!

lib/fog/aws/cdn.rb

@@ -9,7 +9,11 @@ module Fog
       requires :aws_access_key_id, :aws_secret_access_key
       recognizes :host, :path, :port, :scheme, :version, :persistent, :use_iam_profile, :aws_session_token, :aws_credentials_expire_at
 
-      model_path 'fog/aws/cdn/models'
+      model_path 'fog/aws/models/cdn'
+      model       :distribution
+      collection  :distributions
+      model       :streaming_distribution
+      collection  :streaming_distributions
 
       request_path 'fog/aws/requests/cdn'
       request 'delete_distribution'

lib/fog/aws/models/cdn/distribution.rb

@@ -0,0 +1,93 @@
+require 'fog/core/model'
+require 'fog/aws/models/cdn/invalidations'
+require 'fog/aws/models/cdn/distribution_helper'
+
+module Fog
+  module CDN
+    class AWS
+
+      class Distribution < Fog::Model
+        include Fog::CDN::AWS::DistributionHelper
+
+        identity :id,                 :aliases => 'Id'
+
+        attribute :caller_reference,   :aliases => 'CallerReference'
+        attribute :last_modified_time, :aliases => 'LastModifiedTime'
+        attribute :status,             :aliases => 'Status'
+        attribute :s3_origin,          :aliases => 'S3Origin'
+        attribute :custom_origin,      :aliases => 'CustomOrigin'
+        attribute :cname,              :aliases => 'CNAME'
+        attribute :comment,            :aliases => 'Comment'
+        attribute :enabled,            :aliases => 'Enabled'
+        attribute :in_progress_invalidation_batches, :aliases => 'InProgressInvalidationBatches'
+        attribute :logging,            :aliases => 'Logging'
+        attribute :trusted_signers,    :aliases => 'TrustedSigners'
+        attribute :default_root_object,:aliases => 'DefaultRootObject'
+        attribute :domain,             :aliases => 'DomainName'
+        attribute :etag,               :aliases => ['Etag', 'ETag']
+
+        # items part of DistributionConfig
+        CONFIG = [ :caller_reference, :origin,  :cname, :comment, :enabled, :logging, :trusted_signers, :default_root_object ]
+
+        def initialize(new_attributes = {})
+          super(distribution_config_to_attributes(new_attributes))
+        end
+
+        def invalidations
+          @invalidations ||= begin
+            Fog::CDN::AWS::Invalidations.new(
+              :distribution => self,
+              :connection => connection
+            )
+          end
+        end
+
+        def save
+          requires_one :s3_origin, :custom_origin
+          options = attributes_to_options
+          response = identity ? put_distribution_config(identity, etag, options) : post_distribution(options)
+          etag = response.headers['ETag']
+          merge_attributes(response.body)
+          true
+        end
+
+        private
+
+        def delete_distribution(identity, etag)
+          connection.delete_distribution(identity, etag)
+        end
+
+        def put_distribution_config(identity, etag, options)
+          connection.put_distribution_config(identity, etag, options)
+        end
+
+        def post_distribution(options = {})
+          connection.post_distribution(options)
+        end
+
+        def attributes_to_options
+          options = {
+            'CallerReference' => caller_reference,
+            'S3Origin' => s3_origin,
+            'CustomOrigin' => custom_origin,
+            'CNAME' => cname,
+            'Comment' => comment,
+            'Enabled' => enabled,
+            'Logging' => logging,
+            'TrustedSigners' => trusted_signers,
+            'DefaultRootObject' => default_root_object
+          }
+          options.reject! { |k,v| v.nil? }
+          options.reject! { |k,v| v.respond_to?(:empty?) && v.empty? }
+          options
+        end
+
+        def distribution_config_to_attributes(new_attributes = {})
+          new_attributes.merge(new_attributes.delete('DistributionConfig') || {})
+        end
+
+      end
+
+    end
+  end
+end

lib/fog/aws/models/cdn/distribution_helper.rb

@@ -0,0 +1,64 @@
+require 'fog/core/collection'
+
+module Fog
+  module CDN
+    class AWS
+
+      module DistributionHelper
+
+        def destroy
+          requires :identity, :etag, :caller_reference
+          raise "Distribution must be disabled to be deleted" unless disabled?
+          delete_distribution(identity, etag)
+          true
+        end
+
+        def enabled?
+          requires :identity
+          !!enabled and ready?
+        end
+
+        def disabled?
+          requires :identity
+          not enabled? and ready?
+        end
+
+        def custom_origin?
+          requires :identity
+          not custom_origin.nil?
+        end
+
+        def ready?
+          requires :identity
+          status == 'Deployed'
+        end
+
+        def enable
+          requires :identity
+          reload if etag.nil? or caller_reference.nil?
+          unless enabled?
+            self.enabled = true
+            response = put_distribution_config(identity, etag, attributes_to_options)
+            etag = response.headers['ETag']
+            merge_attributes(response.body)
+          end
+          true
+        end
+
+        def disable
+          requires :identity
+          reload if etag.nil? or caller_reference.nil?
+          if enabled?
+            self.enabled = false
+            response = put_distribution_config(identity, etag, attributes_to_options)
+            etag = response.headers['ETag']
+            merge_attributes(response.body)
+          end
+          true
+        end
+
+      end
+
+    end
+  end
+end

lib/fog/aws/models/cdn/distributions.rb

@@ -0,0 +1,32 @@
+require 'fog/core/collection'
+require 'fog/aws/models/cdn/distribution'
+require 'fog/aws/models/cdn/distributions_helper'
+
+module Fog
+  module CDN
+    class AWS
+
+      class Distributions < Fog::Collection
+        include Fog::CDN::AWS::DistributionsHelper
+
+        model Fog::CDN::AWS::Distribution
+
+        attribute :marker,    :aliases => 'Marker'
+        attribute :max_items, :aliases => 'MaxItems'
+        attribute :is_truncated,    :aliases => 'IsTruncated'
+
+        def get_distribution(dist_id)
+          connection.get_distribution(dist_id)
+        end
+
+        def list_distributions(options = {})
+          connection.get_distribution_list(options)
+        end
+
+        alias :each_distribution_this_page :each
+
+      end
+
+    end
+  end
+end

lib/fog/aws/models/cdn/distributions_helper.rb

@@ -0,0 +1,47 @@
+require 'fog/core/collection'
+
+module Fog
+  module CDN
+    class AWS
+
+      module DistributionsHelper
+        def all(options = {})
+          merge_attributes(options)
+          data = list_distributions(options).body
+          merge_attributes('IsTruncated' => data['IsTruncated'], 'Marker' => data['Marker'])
+          if summary = data['DistributionSummary']
+            load(summary.map { |a| { 'DistributionConfig' => a } })
+          else
+            load((data['StreamingDistributionSummary'] || {}).map { |a| { 'StreamingDistributionConfig' => a }})
+          end
+        end
+
+        def get(dist_id)
+          response = get_distribution(dist_id)
+          data = response.body.merge({'ETag' => response.headers['ETag']})
+          new(data)
+        rescue Excon::Errors::NotFound
+          nil
+        end
+
+        def each
+          if !block_given?
+            self
+          else
+            subset = dup.all
+
+            subset.each_distribution_this_page {|f| yield f}
+            while subset.is_truncated
+              subset = subset.all('Marker' => subset.marker, 'MaxItems' => 1000)
+              subset.each_distribution_this_page {|f| yield f}
+            end
+
+            self
+          end
+        end
+
+      end
+
+    end
+  end
+end

lib/fog/aws/models/cdn/invalidation.rb

@@ -0,0 +1,59 @@
+require 'fog/core/model'
+
+module Fog
+  module CDN
+    class AWS
+
+      class Invalidation < Fog::Model
+
+        identity :id,                :aliases => 'Id'
+
+        attribute :status,           :aliases => 'Status'
+        attribute :create_time,      :aliases => 'CreateTime'
+        attribute :caller_reference, :aliases => 'CallerReference'
+        attribute :paths,            :aliases => 'Paths'
+
+        def initialize(new_attributes={})
+          super(invalidation_to_attributes(new_attributes))
+        end
+
+        def distribution
+          @distribution
+        end
+
+        def ready?
+          requires :id, :status
+          status == 'Completed'
+        end
+
+        def save
+          requires :paths, :caller_reference
+          raise "Submitted invalidation cannot be submitted again" if identity
+          response = connection.post_invalidation(distribution.identity, paths, caller_reference || Time.now.to_i.to_s)
+          merge_attributes(invalidation_to_attributes(response.body))
+          true
+        end
+
+        def destroy
+          # invalidations can't be removed, but tests are requiring they do :)
+          true
+        end
+
+        private
+
+        def distribution=(dist)
+          @distribution = dist
+        end
+
+        def invalidation_to_attributes(new_attributes={})
+          invalidation_batch = new_attributes.delete('InvalidationBatch') || {}
+          new_attributes['Paths'] = invalidation_batch['Path']
+          new_attributes['CallerReference'] = invalidation_batch['CallerReference']
+          new_attributes
+        end
+
+      end
+
+    end
+  end
+end

lib/fog/aws/models/cdn/invalidations.rb

@@ -0,0 +1,54 @@
+require 'fog/core/collection'
+require 'fog/aws/models/cdn/invalidation'
+
+module Fog
+  module CDN
+    class AWS
+
+      class Invalidations < Fog::Collection
+
+        attribute :is_truncated,            :aliases => ['IsTruncated']
+        attribute :max_items,               :aliases => ['MaxItems']
+        attribute :next_marker,             :aliases => ['NextMarker']
+        attribute :marker,                  :aliases => ['Marker']
+
+        attribute :distribution
+
+        model Fog::CDN::AWS::Invalidation
+
+        def all(options = {})
+          requires :distribution
+          options[:max_items]  ||= max_items
+          options.delete_if {|key, value| value.nil?}
+
+          data = connection.get_invalidation_list(distribution.identity, options).body
+
+          merge_attributes(data.reject {|key, value| !['IsTruncated', 'MaxItems', 'NextMarker', 'Marker'].include?(key)})
+
+          load(data['InvalidationSummary'])
+        end
+
+        def get(invalidation_id)
+          requires :distribution
+
+          data = connection.get_invalidation(distribution.identity, invalidation_id).body
+
+          if data
+            invalidation = new(data)
+          else
+            nil
+          end
+        rescue Excon::Errors::NotFound
+          nil
+        end
+
+        def new(attributes = {})
+          requires :distribution
+          super({ :distribution => distribution }.merge!(attributes))
+        end
+
+      end
+
+    end
+  end
+end

lib/fog/aws/models/cdn/streaming_distribution.rb

@@ -0,0 +1,77 @@
+require 'fog/core/model'
+require 'fog/aws/models/cdn/invalidations'
+require 'fog/aws/models/cdn/distribution_helper'
+
+module Fog
+  module CDN
+    class AWS
+
+      class StreamingDistribution < Fog::Model
+        include Fog::CDN::AWS::DistributionHelper
+
+        identity :id,                 :aliases => 'Id'
+
+        attribute :caller_reference,   :aliases => 'CallerReference'
+        attribute :last_modified_time, :aliases => 'LastModifiedTime'
+        attribute :status,             :aliases => 'Status'
+        attribute :s3_origin,          :aliases => 'S3Origin'
+        attribute :cname,              :aliases => 'CNAME'
+        attribute :comment,            :aliases => 'Comment'
+        attribute :enabled,            :aliases => 'Enabled'
+        attribute :logging,            :aliases => 'Logging'
+        attribute :domain,             :aliases => 'DomainName'
+        attribute :etag,               :aliases => ['Etag', 'ETag']
+
+        # items part of DistributionConfig
+        CONFIG = [ :caller_reference, :cname, :comment, :enabled, :logging ]
+
+        def initialize(new_attributes = {})
+          super(distribution_config_to_attributes(new_attributes))
+        end
+
+        def save
+          requires_one :s3_origin
+          options = attributes_to_options
+          response = identity ? put_distribution_config(identity, etag, options) : post_distribution(options)
+          etag = response.headers['ETag']
+          merge_attributes(response.body)
+          true
+        end
+
+        private
+
+        def delete_distribution(identity, etag)
+          connection.delete_streaming_distribution(identity, etag)
+        end
+
+        def put_distribution_config(identity, etag, options)
+          connection.put_streaming_distribution_config(identity, etag, options)
+        end
+
+        def post_distribution(options = {})
+          connection.post_streaming_distribution(options)
+        end
+
+        def attributes_to_options
+          options = {
+            'CallerReference' => caller_reference,
+            'S3Origin' => s3_origin,
+            'CNAME' => cname,
+            'Comment' => comment,
+            'Enabled' => enabled,
+            'Logging' => logging,
+          }
+          options.reject! { |k,v| v.nil? }
+          options.reject! { |k,v| v.respond_to?(:empty?) && v.empty? }
+          options
+        end
+
+        def distribution_config_to_attributes(new_attributes = {})
+          new_attributes.merge(new_attributes.delete('StreamingDistributionConfig') || {})
+        end
+
+      end
+
+    end
+  end
+end

lib/fog/aws/models/cdn/streaming_distributions.rb

@@ -0,0 +1,32 @@
+require 'fog/core/collection'
+require 'fog/aws/models/cdn/streaming_distribution'
+require 'fog/aws/models/cdn/distributions_helper'
+
+module Fog
+  module CDN
+    class AWS
+
+      class StreamingDistributions < Fog::Collection
+        include Fog::CDN::AWS::DistributionsHelper
+
+        model Fog::CDN::AWS::StreamingDistribution
+
+        attribute :marker,    :aliases => 'Marker'
+        attribute :max_items, :aliases => 'MaxItems'
+        attribute :is_truncated,    :aliases => 'IsTruncated'
+
+        def get_distribution(dist_id)
+          connection.get_streaming_distribution(dist_id)
+        end
+
+        def list_distributions(options = {})
+          connection.get_streaming_distribution_list(options)
+        end
+
+        alias :each_distribution_this_page :each
+
+      end
+
+    end
+  end
+end

tests/aws/models/cdn/distribution_tests.rb

@@ -0,0 +1,19 @@
+Shindo.tests("Fog::CDN[:aws] | distribution", ['aws', 'cdn']) do
+  params = { :s3_origin => { 'DNSName' => 'fog_test_cdn.s3.amazonaws.com'}, :enabled => true }
+  model_tests(Fog::CDN[:aws].distributions, params, false) do
+    # distribution needs to be ready before being disabled
+    tests("#ready? - may take 15 minutes to complete...").succeeds do
+      pending if Fog.mocking?
+
+      @instance.wait_for { ready? }
+    end
+
+    # and disabled before being distroyed
+    tests("#disable - may take 15 minutes to complete...").succeeds do
+      pending if Fog.mocking?
+
+      @instance.disable
+      @instance.wait_for { ready? }
+    end
+  end
+end

tests/aws/models/cdn/distributions_tests.rb

@@ -0,0 +1,19 @@
+Shindo.tests("Fog::CDN[:aws] | distributions", ['aws', 'cdn']) do
+  params = { :s3_origin => { 'DNSName' => 'fog_test_cdn.s3.amazonaws.com'}, :enabled => true}
+  collection_tests(Fog::CDN[:aws].distributions, params, false) do
+    # distribution needs to be ready before being disabled
+    tests("#ready? - may take 15 minutes to complete...").succeeds do
+      pending if Fog.mocking?
+
+      @instance.wait_for { ready? }
+    end
+
+    # and disabled before being distroyed
+    tests("#disable - may take 15 minutes to complete...").succeeds do
+      pending if Fog.mocking?
+
+      @instance.disable
+      @instance.wait_for { ready? }
+    end
+  end
+end

tests/aws/models/cdn/invalidation_tests.rb

@@ -0,0 +1,34 @@
+Shindo.tests("Fog::CDN[:aws] | invalidation", ['aws', 'cdn']) do
+
+  pending if Fog.mocking?
+
+  tests("distributions#create").succeeds do
+    @distribution = Fog::CDN[:aws].distributions.create(:s3_origin => {'DNSName' => 'fog_test.s3.amazonaws.com'}, :enabled => true)
+  end
+
+  params = { :paths => [ '/index.html', '/path/to/index.html' ] }
+
+  model_tests(@distribution.invalidations, params, false) do
+
+    tests("#id") do
+      returns(true) { @instance.identity != nil }
+    end
+
+    tests("#paths") do
+      returns([ '/index.html', '/path/to/index.html' ].sort) { @instance.paths.sort }
+    end
+
+    tests("#ready? - may take 15 minutes to complete...").succeeds do
+      @instance.wait_for { ready? }
+    end
+  end
+
+  tests("distribution#destroy - may take around 15/20 minutes to complete...").succeeds do
+    @distribution.wait_for { ready? }
+    @distribution.disable
+    @distribution.wait_for { ready? }
+    @distribution.destroy
+  end
+
+end
+

tests/aws/models/cdn/invalidations_tests.rb

@@ -0,0 +1,17 @@
+Shindo.tests("Fog::CDN[:aws] | invalidations", ['aws', 'cdn']) do
+  pending if Fog.mocking?
+
+  tests("distributions#create").succeeds do
+    @distribution = Fog::CDN[:aws].distributions.create(:s3_origin => {'DNSName' => 'fog_test.s3.amazonaws.com'}, :enabled => true)
+  end
+
+  collection_tests(@distribution.invalidations, { :paths => [ '/index.html' ]}, false)
+
+  tests("distribution#destroy - may take 15/20 minutes to complete").succeeds do
+    @distribution.wait_for { ready? }
+    @distribution.disable
+    @distribution.wait_for { ready? }
+    @distribution.destroy
+  end
+end
+

tests/aws/models/cdn/streaming_distribution_tests.rb

@@ -0,0 +1,19 @@
+Shindo.tests("Fog::CDN[:aws] | streaming_distribution", ['aws', 'cdn']) do
+  params = { :s3_origin => { 'DNSName' => 'fog_test_cdn.s3.amazonaws.com'}, :enabled => true }
+  model_tests(Fog::CDN[:aws].streaming_distributions, params, false) do
+    # distribution needs to be ready before being disabled
+    tests("#ready? - may take 15 minutes to complete...").succeeds do
+      pending if Fog.mocking?
+
+      @instance.wait_for { ready? }
+    end
+
+    # and disabled before being distroyed
+    tests("#disable - may take 15 minutes to complete...").succeeds do
+      pending if Fog.mocking?
+
+      @instance.disable
+      @instance.wait_for { ready? }
+    end
+  end
+end

tests/aws/models/cdn/streaming_distributions_tests.rb

@@ -0,0 +1,19 @@
+Shindo.tests("Fog::CDN[:aws] | streaming_distributions", ['aws', 'cdn']) do
+  params = { :s3_origin => { 'DNSName' => 'fog_test_cdn.s3.amazonaws.com'}, :enabled => true}
+  collection_tests(Fog::CDN[:aws].streaming_distributions, params, false) do
+    # distribution needs to be ready before being disabled
+    tests("#ready? - may take 15 minutes to complete...").succeeds do
+      pending if Fog.mocking?
+
+      @instance.wait_for { ready? }
+    end
+
+    # and disabled before being distroyed
+    tests("#disable - may take 15 minutes to complete...").succeeds do
+      pending if Fog.mocking?
+
+      @instance.disable
+      @instance.wait_for { ready? }
+    end
+  end
+end