fog--fog/lib/fog/openstack/docs/storage.md

Storage

This document explains how to get started using OpenStack Swift with Fog.

Starting irb console

Start by executing the following command:

irb

Once irb has launched you need to require the Fog library.

If using Ruby 1.8.x execute:

require 'rubygems'
require 'fog'

If using Ruby 1.9.x execute:

require 'fog'

Create Service

Next, create a connection to Swift:

service = Fog::Storage.new({
  :provider            => 'OpenStack',   # OpenStack Fog provider
  :openstack_username  => USERNAME,      # Your OpenStack Username
  :openstack_api_key   => PASSWORD,      # Your OpenStack Password
  :openstack_auth_url  => 'http://YOUR_OPENSTACK_ENDPOINT:PORT/v2.0/tokens'
})

Alternative regions are specified using the key :openstack_region . A list of regions available for Swift can be found by executing the following:

Optional Service Parameters

The Storage service supports the following additional parameters:

Key	Description
:persistent	If set to true, the service will use a persistent connection.
:openstack_service_name
:openstack_service_type
:openstack_tenant
:openstack_region
:openstack_temp_url_key

Optional Connection Parameters

Fog supports passing additional connection parameters to its underlying HTTP library (Excon) using the :connection_options parameter.

Key	Description
:connect_timeout	Connection timeout (default: 60 seconds)
:write_timeout	Write timeout for connection (default: 60 seconds)
:proxy	Proxy for HTTP and HTTPS connections
:ssl_ca_path	Path to SSL certificate authorities
:ssl_ca_file	SSL certificate authority file
:ssl_verify_peer	SSL verify peer (default: true)

Fog Abstractions

Fog provides both a model and request abstraction. The request abstraction provides the most efficient interface and the model abstraction wraps the request abstraction to provide a convenient ActiveModel like interface.

Request Layer

The Fog::Storage object supports a number of methods that wrap individual HTTP requests to the Swift API.

To see a list of requests supported by the storage service:

service.requests

This returns:

[:copy_object, :delete_container, :delete_object, :delete_multiple_objects, :delete_static_large_object, :get_container, :get_containers, :get_object, :get_object_http_url, :get_object_https_url, :head_container, :head_containers, :head_object, :put_container, :put_object, :put_object_manifest, :put_dynamic_obj_manifest, :put_static_obj_manifest, :post_set_meta_temp_url_key]

Example Request

To request a view account details:

response = service.head_containers

This returns in the following Excon::Response:

#<Excon::Response:0x10283fc68 @headers={"X-Account-Bytes-Used"=>"2563554", "Date"=>"Thu, 21 Feb 2013 21:57:02 GMT", "X-Account-Meta-Temp-Url-Key"=>"super_secret_key", "X-Timestamp"=>"1354552916.82056", "Content-Length"=>"0", "Content-Type"=>"application/json; charset=utf-8", "X-Trans-Id"=>"txe934924374a744c8a6c40dd8f29ab94a", "Accept-Ranges"=>"bytes", "X-Account-Container-Count"=>"7", "X-Account-Object-Count"=>"5"}, @status=204, @body="">

To view the status of the response:

response.status

Note: Fog is aware of the valid HTTP response statuses for each request type. If an unexpected HTTP response status occurs, Fog will raise an exception.

To view response headers:

response.headers

This will return:

{"X-Account-Bytes-Used"=>"2563554", "Date"=>"Thu, 21 Feb 2013 21:57:02 GMT", "X-Account-Meta-Temp-Url-Key"=>"super_secret_key", "X-Timestamp"=>"1354552916.82056", "Content-Length"=>"0", "Content-Type"=>"application/json; charset=utf-8", "X-Trans-Id"=>"txe934924374a744c8a6c40dd8f29ab94a", "Accept-Ranges"=>"bytes", "X-Account-Container-Count"=>"7", "X-Account-Object-Count"=>"5"}

To learn more about Fog::Storage request methods refer to rdoc. To learn more about Excon refer to Excon GitHub repo.

Model Layer

Fog models behave in a manner similar to ActiveModel. Models will generally respond to create, save, destroy, reload and attributes methods. Additionally, fog will automatically create attribute accessors.

Here is a summary of common model methods:

Method	Description
create	Accepts hash of attributes and creates object. Note: creation is a non-blocking call and you will be required to wait for a valid state before using resulting object.
save	Saves object. Note: not all objects support updating object.
destroy	Destroys object. Note: this is a non-blocking call and object deletion might not be instantaneous.
reload	Updates object with latest state from service.
attributes	Returns a hash containing the list of model attributes and values.
identity	Returns the identity of the object. Note: This might not always be equal to object.id.

The remainder of this document details the model abstraction.

Note: Fog sometimes refers to Swift containers as directories.

List Directories

To retrieve a list of directories:

service.directories

This returns a collection of Fog::Storage::OpenStack::Directory models:

Get Directory

To retrieve a specific directory:

service.directories.get "blue"

This returns a Fog::Storage::OpenStack::Directory instance:

Create Directory

To create a directory:

service.directories.create :key => 'backups'

Additional Parameters

The create method also supports the following key values:

Key	Description
:metadata	Hash containing directory metadata.

Delete Directory

To delete a directory:

directory.destroy

Note: Directory must be empty before it can be deleted.

Directory URL

To get a directory's URL:

directory.public_url

List Files

To list files in a directory:

directory.files

Note: File contents is not downloaded until body attribute is called.

Upload Files

To upload a file into a directory:

file = directory.files.create :key => 'space.jpg', :body => File.open "space.jpg"

Note: For files larger than 5 GB please refer to the Upload Large Files section.

Additional Parameters

The create method also supports the following key values:

Key	Description
:content_type	The content type of the object. Cloud Files will attempt to auto detect this value if omitted.
:access_control_allow_origin	URLs can make Cross Origin Requests. Format is http://www.example.com. Separate URLs with a space. An asterisk (*) allows all. Please refer to CORS Container Headers for more information.
:origin	The origin is the URI of the object's host.
:etag	The MD5 checksum of your object's data. If specified, Cloud Files will validate the integrity of the uploaded object.
:metadata	Hash containing file metadata.

Upload Large Files

Swift requires files larger than 5 GB (the Swift default limit) to be uploaded into segments along with an accompanying manifest file. All of the segments must be uploaded to the same container.

	SEGMENT_LIMIT = 5368709119.0  # 5GB -1
	BUFFER_SIZE = 1024 * 1024 # 1MB

	File.open(file_name) do |f|
	  segment = 0
	  until file.eof?
	    segment += 1
	    offset = 0

	    # upload segment to cloud files
	    segment_suffix = segment.to_s.rjust(10, '0')
	    service.put_object("my_container", "large_file/#{segment_suffix}", nil) do
	      if offset <= SEGMENT_LIMIT - BUFFER_SIZE
	        buf = file.read(BUFFER_SIZE).to_s
	        offset += buf.size
	        buf
	      else
	        ''
	      end
	    end
	  end
	end

	# write manifest file
	service.put_object_manifest("my_container", "large_file", 'X-Object-Manifest' => "my_container/large_file/")

    # write manifest file
    service.put_object_manifest("my_container", "large_file", 'X-Object-Manifest' => "my_container/large_file/")

    # write manifest file
    service.put_object_manifest("my_container", "large_file", 'X-Object-Manifest' => "my_container/large_file/")

Segmented files are downloaded like ordinary files. See Download Files section for more information.

Download Files

The most efficient way to download files from a private or public directory is as follows:

File.open('downloaded-file.jpg', 'w') do | f |
  directory.files.get("my_big_file.jpg") do | data, remaining, content_length |
    f.syswrite data
  end
end

This will download and save the file in 1 MB chunks. The chunk size can be changed by passing the parameter :chunk_size into the :connection_options hash in the service constructor.

Note: The body attribute of file will be empty if a file has been downloaded using this method.

If a file object has already been loaded into memory, you can save it as follows:

File.open('germany.jpg', 'w') {|f| f.write(file_object.body) }

Note: This method is more memory intensive as the entire object is loaded into memory before saving the file as in the example above.

File URL

To get a file's URL:

file.public_url

Metadata

You can access metadata as an attribute on Fog::Storage::Rackspace::File.

file.metadata[:environment]

File metadata is set when the file is saved:

file.save

Metadata is reloaded when directory or file is reloaded:

file.reload

Copy File

Cloud Files supports copying files. To copy files into a container named "trip" with a name of "europe.jpg" do the following:

file.copy("trip", "europe.jpg")

To move or rename a file, perform a copy operation and then delete the old file:

file.copy("trip", "germany.jpg")
file.destroy

Delete File

To delete a file:

file.destroy

Additional Resources

• fog.io
• Fog rdoc
• Fog Github repo
• Fog Github Issues
• Excon Github repo
• Swift API

Support and Feedback

Your feedback is appreciated! If you have specific issues with the fog SDK, you should file an issue via Github.