[aws|cdn] AWS CDN models

This patch brings models for AWS CDN, including tests and documentation.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>
This commit is contained in:
Brice Figureau 2012-09-26 18:51:00 +02:00
parent 7a7bc53a89
commit 9666c9c7e0
16 changed files with 656 additions and 1 deletions

View File

@ -46,12 +46,63 @@ Now you'll need to create a 'distribution' which represents a mapping from the C
cdn.get_distribution(distribution_id).body['Status'] == 'Deployed'
}
Fog also supports models for the AWS CDN. The above code can also be written like this:
distribution = cdn.distributions.create( :custom_origin => {
'DNSName' => 'www.example.com',
'OriginProtocolPolicy' => 'match-viewer'
}, :enabled => true
})
distribution.wait_for { ready? }
Like other collections supported by Fog, it is also possible to browse the list of distributions:
cdn.distributions.all
Or get access to a distinct distribution by its identity:
cdn.distributions.get(distribution_id)
## Getting Served
With the domain name from the distribution in hand you should now be ready to serve content from the edge. All you need to do is start replacing urls like `http://www.example.com/stylesheets/foo.css` with `#{cdn_domain_name}/stylesheets/foo.css`. Just because you can do something doesn't always mean you should though. Dynamic pages are not really well suited to CDN storage, since CDN content will be the same for every user. Fortunately some of your most used content is a great fit. By just switching over your images, javascripts and stylesheets you can have an impact for each and every one of your users.
Congrats, your site is faster! By default the urls aren't very pretty, something like `http://d1xdx2sah5udd0.cloudfront.net/stylesheets/foo.css`. Thankfully you can use CNAME config options to utilize something like `http://assets.example.com/stylesheets/foo.css`, if you are interested in learning more about this let me know in the comments.
## Invalidating the CDN caches
Sometimes, some part of the CDN cache needs to be invalidated because the origin changed and we need a faster propagation than waiting for the objects to expire by themselves. To do this, CloudFront supports creating <a href="http://docs.amazonwebservices.com/AmazonCloudFront/latest/APIReference/Actions_Invalidations.html">distributions invalidation</a>.
An invalidation can be created with the following code:
# let's invalidate /test.html and /path/to/file.html
data = cdn.post_invalidation(distribution_id, [ "/test.html", "/path/to/file.html" ])
invalidation_id = data.body['Id']
Fog.wait_for { cdn.get_invalidation(distribution_id, invalidation_id).body['Status'] == 'Completed' }
It is also possible to list past and current invalidation for a given distribution:
cdn.get_invalidation_list(distribution_id)
The same can be written with Fog CDN model abstraction:
distribution = cdn.distributions.get(distribution_id)
invalidation = distribution.invalidations.create(:paths => [ "/test.html", "/path/to/file.html" ])
invalidation.wait_for { ready? }
Listing invalidations is as simple as:
distribution.invalidations.all
# this returns only summarized invalidation
# to get access to the invalidations path:
distribution.invalidations.get(invalidation_id)
## Cleaning Up
But, just in case you need to update things I'll run through how you can make changes. In my case I just want to clean up after myself, so I'll use the distribution_id and ETag from before to disable the distribution. We need to use the ETag as well because it provides a way to refer to different versions of the same distribution and ensures we are updating the version that we think we are.
@ -79,4 +130,19 @@ Now you just need to wait for the update to happen like before and once its disa
}
cdn.delete_distribution(distribution_id, etag)
This can also be written with CDN models as:
distribution = cdn.distributions.get(distribution_id)
# make sure the distribution is deployed otherwise it can't be disabled
distribution.wait_for { ready? }
distribution.disable
# Disabling a distribution is a lengthy operation
distribution.wait_for { ready? }
# and finally let's get rid of it
distribution.destroy
Thats it, now go forth and speed up some load times!

View File

@ -9,7 +9,11 @@ module Fog
requires :aws_access_key_id, :aws_secret_access_key
recognizes :host, :path, :port, :scheme, :version, :persistent, :use_iam_profile, :aws_session_token, :aws_credentials_expire_at
model_path 'fog/aws/cdn/models'
model_path 'fog/aws/models/cdn'
model :distribution
collection :distributions
model :streaming_distribution
collection :streaming_distributions
request_path 'fog/aws/requests/cdn'
request 'delete_distribution'

View File

@ -0,0 +1,93 @@
require 'fog/core/model'
require 'fog/aws/models/cdn/invalidations'
require 'fog/aws/models/cdn/distribution_helper'
module Fog
module CDN
class AWS
class Distribution < Fog::Model
include Fog::CDN::AWS::DistributionHelper
identity :id, :aliases => 'Id'
attribute :caller_reference, :aliases => 'CallerReference'
attribute :last_modified_time, :aliases => 'LastModifiedTime'
attribute :status, :aliases => 'Status'
attribute :s3_origin, :aliases => 'S3Origin'
attribute :custom_origin, :aliases => 'CustomOrigin'
attribute :cname, :aliases => 'CNAME'
attribute :comment, :aliases => 'Comment'
attribute :enabled, :aliases => 'Enabled'
attribute :in_progress_invalidation_batches, :aliases => 'InProgressInvalidationBatches'
attribute :logging, :aliases => 'Logging'
attribute :trusted_signers, :aliases => 'TrustedSigners'
attribute :default_root_object,:aliases => 'DefaultRootObject'
attribute :domain, :aliases => 'DomainName'
attribute :etag, :aliases => ['Etag', 'ETag']
# items part of DistributionConfig
CONFIG = [ :caller_reference, :origin, :cname, :comment, :enabled, :logging, :trusted_signers, :default_root_object ]
def initialize(new_attributes = {})
super(distribution_config_to_attributes(new_attributes))
end
def invalidations
@invalidations ||= begin
Fog::CDN::AWS::Invalidations.new(
:distribution => self,
:connection => connection
)
end
end
def save
requires_one :s3_origin, :custom_origin
options = attributes_to_options
response = identity ? put_distribution_config(identity, etag, options) : post_distribution(options)
etag = response.headers['ETag']
merge_attributes(response.body)
true
end
private
def delete_distribution(identity, etag)
connection.delete_distribution(identity, etag)
end
def put_distribution_config(identity, etag, options)
connection.put_distribution_config(identity, etag, options)
end
def post_distribution(options = {})
connection.post_distribution(options)
end
def attributes_to_options
options = {
'CallerReference' => caller_reference,
'S3Origin' => s3_origin,
'CustomOrigin' => custom_origin,
'CNAME' => cname,
'Comment' => comment,
'Enabled' => enabled,
'Logging' => logging,
'TrustedSigners' => trusted_signers,
'DefaultRootObject' => default_root_object
}
options.reject! { |k,v| v.nil? }
options.reject! { |k,v| v.respond_to?(:empty?) && v.empty? }
options
end
def distribution_config_to_attributes(new_attributes = {})
new_attributes.merge(new_attributes.delete('DistributionConfig') || {})
end
end
end
end
end

View File

@ -0,0 +1,64 @@
require 'fog/core/collection'
module Fog
module CDN
class AWS
module DistributionHelper
def destroy
requires :identity, :etag, :caller_reference
raise "Distribution must be disabled to be deleted" unless disabled?
delete_distribution(identity, etag)
true
end
def enabled?
requires :identity
!!enabled and ready?
end
def disabled?
requires :identity
not enabled? and ready?
end
def custom_origin?
requires :identity
not custom_origin.nil?
end
def ready?
requires :identity
status == 'Deployed'
end
def enable
requires :identity
reload if etag.nil? or caller_reference.nil?
unless enabled?
self.enabled = true
response = put_distribution_config(identity, etag, attributes_to_options)
etag = response.headers['ETag']
merge_attributes(response.body)
end
true
end
def disable
requires :identity
reload if etag.nil? or caller_reference.nil?
if enabled?
self.enabled = false
response = put_distribution_config(identity, etag, attributes_to_options)
etag = response.headers['ETag']
merge_attributes(response.body)
end
true
end
end
end
end
end

View File

@ -0,0 +1,32 @@
require 'fog/core/collection'
require 'fog/aws/models/cdn/distribution'
require 'fog/aws/models/cdn/distributions_helper'
module Fog
module CDN
class AWS
class Distributions < Fog::Collection
include Fog::CDN::AWS::DistributionsHelper
model Fog::CDN::AWS::Distribution
attribute :marker, :aliases => 'Marker'
attribute :max_items, :aliases => 'MaxItems'
attribute :is_truncated, :aliases => 'IsTruncated'
def get_distribution(dist_id)
connection.get_distribution(dist_id)
end
def list_distributions(options = {})
connection.get_distribution_list(options)
end
alias :each_distribution_this_page :each
end
end
end
end

View File

@ -0,0 +1,47 @@
require 'fog/core/collection'
module Fog
module CDN
class AWS
module DistributionsHelper
def all(options = {})
merge_attributes(options)
data = list_distributions(options).body
merge_attributes('IsTruncated' => data['IsTruncated'], 'Marker' => data['Marker'])
if summary = data['DistributionSummary']
load(summary.map { |a| { 'DistributionConfig' => a } })
else
load((data['StreamingDistributionSummary'] || {}).map { |a| { 'StreamingDistributionConfig' => a }})
end
end
def get(dist_id)
response = get_distribution(dist_id)
data = response.body.merge({'ETag' => response.headers['ETag']})
new(data)
rescue Excon::Errors::NotFound
nil
end
def each
if !block_given?
self
else
subset = dup.all
subset.each_distribution_this_page {|f| yield f}
while subset.is_truncated
subset = subset.all('Marker' => subset.marker, 'MaxItems' => 1000)
subset.each_distribution_this_page {|f| yield f}
end
self
end
end
end
end
end
end

View File

@ -0,0 +1,59 @@
require 'fog/core/model'
module Fog
module CDN
class AWS
class Invalidation < Fog::Model
identity :id, :aliases => 'Id'
attribute :status, :aliases => 'Status'
attribute :create_time, :aliases => 'CreateTime'
attribute :caller_reference, :aliases => 'CallerReference'
attribute :paths, :aliases => 'Paths'
def initialize(new_attributes={})
super(invalidation_to_attributes(new_attributes))
end
def distribution
@distribution
end
def ready?
requires :id, :status
status == 'Completed'
end
def save
requires :paths, :caller_reference
raise "Submitted invalidation cannot be submitted again" if identity
response = connection.post_invalidation(distribution.identity, paths, caller_reference || Time.now.to_i.to_s)
merge_attributes(invalidation_to_attributes(response.body))
true
end
def destroy
# invalidations can't be removed, but tests are requiring they do :)
true
end
private
def distribution=(dist)
@distribution = dist
end
def invalidation_to_attributes(new_attributes={})
invalidation_batch = new_attributes.delete('InvalidationBatch') || {}
new_attributes['Paths'] = invalidation_batch['Path']
new_attributes['CallerReference'] = invalidation_batch['CallerReference']
new_attributes
end
end
end
end
end

View File

@ -0,0 +1,54 @@
require 'fog/core/collection'
require 'fog/aws/models/cdn/invalidation'
module Fog
module CDN
class AWS
class Invalidations < Fog::Collection
attribute :is_truncated, :aliases => ['IsTruncated']
attribute :max_items, :aliases => ['MaxItems']
attribute :next_marker, :aliases => ['NextMarker']
attribute :marker, :aliases => ['Marker']
attribute :distribution
model Fog::CDN::AWS::Invalidation
def all(options = {})
requires :distribution
options[:max_items] ||= max_items
options.delete_if {|key, value| value.nil?}
data = connection.get_invalidation_list(distribution.identity, options).body
merge_attributes(data.reject {|key, value| !['IsTruncated', 'MaxItems', 'NextMarker', 'Marker'].include?(key)})
load(data['InvalidationSummary'])
end
def get(invalidation_id)
requires :distribution
data = connection.get_invalidation(distribution.identity, invalidation_id).body
if data
invalidation = new(data)
else
nil
end
rescue Excon::Errors::NotFound
nil
end
def new(attributes = {})
requires :distribution
super({ :distribution => distribution }.merge!(attributes))
end
end
end
end
end

View File

@ -0,0 +1,77 @@
require 'fog/core/model'
require 'fog/aws/models/cdn/invalidations'
require 'fog/aws/models/cdn/distribution_helper'
module Fog
module CDN
class AWS
class StreamingDistribution < Fog::Model
include Fog::CDN::AWS::DistributionHelper
identity :id, :aliases => 'Id'
attribute :caller_reference, :aliases => 'CallerReference'
attribute :last_modified_time, :aliases => 'LastModifiedTime'
attribute :status, :aliases => 'Status'
attribute :s3_origin, :aliases => 'S3Origin'
attribute :cname, :aliases => 'CNAME'
attribute :comment, :aliases => 'Comment'
attribute :enabled, :aliases => 'Enabled'
attribute :logging, :aliases => 'Logging'
attribute :domain, :aliases => 'DomainName'
attribute :etag, :aliases => ['Etag', 'ETag']
# items part of DistributionConfig
CONFIG = [ :caller_reference, :cname, :comment, :enabled, :logging ]
def initialize(new_attributes = {})
super(distribution_config_to_attributes(new_attributes))
end
def save
requires_one :s3_origin
options = attributes_to_options
response = identity ? put_distribution_config(identity, etag, options) : post_distribution(options)
etag = response.headers['ETag']
merge_attributes(response.body)
true
end
private
def delete_distribution(identity, etag)
connection.delete_streaming_distribution(identity, etag)
end
def put_distribution_config(identity, etag, options)
connection.put_streaming_distribution_config(identity, etag, options)
end
def post_distribution(options = {})
connection.post_streaming_distribution(options)
end
def attributes_to_options
options = {
'CallerReference' => caller_reference,
'S3Origin' => s3_origin,
'CNAME' => cname,
'Comment' => comment,
'Enabled' => enabled,
'Logging' => logging,
}
options.reject! { |k,v| v.nil? }
options.reject! { |k,v| v.respond_to?(:empty?) && v.empty? }
options
end
def distribution_config_to_attributes(new_attributes = {})
new_attributes.merge(new_attributes.delete('StreamingDistributionConfig') || {})
end
end
end
end
end

View File

@ -0,0 +1,32 @@
require 'fog/core/collection'
require 'fog/aws/models/cdn/streaming_distribution'
require 'fog/aws/models/cdn/distributions_helper'
module Fog
module CDN
class AWS
class StreamingDistributions < Fog::Collection
include Fog::CDN::AWS::DistributionsHelper
model Fog::CDN::AWS::StreamingDistribution
attribute :marker, :aliases => 'Marker'
attribute :max_items, :aliases => 'MaxItems'
attribute :is_truncated, :aliases => 'IsTruncated'
def get_distribution(dist_id)
connection.get_streaming_distribution(dist_id)
end
def list_distributions(options = {})
connection.get_streaming_distribution_list(options)
end
alias :each_distribution_this_page :each
end
end
end
end

View File

@ -0,0 +1,19 @@
Shindo.tests("Fog::CDN[:aws] | distribution", ['aws', 'cdn']) do
params = { :s3_origin => { 'DNSName' => 'fog_test_cdn.s3.amazonaws.com'}, :enabled => true }
model_tests(Fog::CDN[:aws].distributions, params, false) do
# distribution needs to be ready before being disabled
tests("#ready? - may take 15 minutes to complete...").succeeds do
pending if Fog.mocking?
@instance.wait_for { ready? }
end
# and disabled before being distroyed
tests("#disable - may take 15 minutes to complete...").succeeds do
pending if Fog.mocking?
@instance.disable
@instance.wait_for { ready? }
end
end
end

View File

@ -0,0 +1,19 @@
Shindo.tests("Fog::CDN[:aws] | distributions", ['aws', 'cdn']) do
params = { :s3_origin => { 'DNSName' => 'fog_test_cdn.s3.amazonaws.com'}, :enabled => true}
collection_tests(Fog::CDN[:aws].distributions, params, false) do
# distribution needs to be ready before being disabled
tests("#ready? - may take 15 minutes to complete...").succeeds do
pending if Fog.mocking?
@instance.wait_for { ready? }
end
# and disabled before being distroyed
tests("#disable - may take 15 minutes to complete...").succeeds do
pending if Fog.mocking?
@instance.disable
@instance.wait_for { ready? }
end
end
end

View File

@ -0,0 +1,34 @@
Shindo.tests("Fog::CDN[:aws] | invalidation", ['aws', 'cdn']) do
pending if Fog.mocking?
tests("distributions#create").succeeds do
@distribution = Fog::CDN[:aws].distributions.create(:s3_origin => {'DNSName' => 'fog_test.s3.amazonaws.com'}, :enabled => true)
end
params = { :paths => [ '/index.html', '/path/to/index.html' ] }
model_tests(@distribution.invalidations, params, false) do
tests("#id") do
returns(true) { @instance.identity != nil }
end
tests("#paths") do
returns([ '/index.html', '/path/to/index.html' ].sort) { @instance.paths.sort }
end
tests("#ready? - may take 15 minutes to complete...").succeeds do
@instance.wait_for { ready? }
end
end
tests("distribution#destroy - may take around 15/20 minutes to complete...").succeeds do
@distribution.wait_for { ready? }
@distribution.disable
@distribution.wait_for { ready? }
@distribution.destroy
end
end

View File

@ -0,0 +1,17 @@
Shindo.tests("Fog::CDN[:aws] | invalidations", ['aws', 'cdn']) do
pending if Fog.mocking?
tests("distributions#create").succeeds do
@distribution = Fog::CDN[:aws].distributions.create(:s3_origin => {'DNSName' => 'fog_test.s3.amazonaws.com'}, :enabled => true)
end
collection_tests(@distribution.invalidations, { :paths => [ '/index.html' ]}, false)
tests("distribution#destroy - may take 15/20 minutes to complete").succeeds do
@distribution.wait_for { ready? }
@distribution.disable
@distribution.wait_for { ready? }
@distribution.destroy
end
end

View File

@ -0,0 +1,19 @@
Shindo.tests("Fog::CDN[:aws] | streaming_distribution", ['aws', 'cdn']) do
params = { :s3_origin => { 'DNSName' => 'fog_test_cdn.s3.amazonaws.com'}, :enabled => true }
model_tests(Fog::CDN[:aws].streaming_distributions, params, false) do
# distribution needs to be ready before being disabled
tests("#ready? - may take 15 minutes to complete...").succeeds do
pending if Fog.mocking?
@instance.wait_for { ready? }
end
# and disabled before being distroyed
tests("#disable - may take 15 minutes to complete...").succeeds do
pending if Fog.mocking?
@instance.disable
@instance.wait_for { ready? }
end
end
end

View File

@ -0,0 +1,19 @@
Shindo.tests("Fog::CDN[:aws] | streaming_distributions", ['aws', 'cdn']) do
params = { :s3_origin => { 'DNSName' => 'fog_test_cdn.s3.amazonaws.com'}, :enabled => true}
collection_tests(Fog::CDN[:aws].streaming_distributions, params, false) do
# distribution needs to be ready before being disabled
tests("#ready? - may take 15 minutes to complete...").succeeds do
pending if Fog.mocking?
@instance.wait_for { ready? }
end
# and disabled before being distroyed
tests("#disable - may take 15 minutes to complete...").succeeds do
pending if Fog.mocking?
@instance.disable
@instance.wait_for { ready? }
end
end
end