kotovalexarian-likes-github/capistrano
1
0
Fork 0
mirror of https://github.com/capistrano/capistrano synced 2023-03-27 23:21:18 -04:00
Code Releases Activity

Refactor and improve Host and Role Filtering

Browse source 
Update CHANGELOG and README
Replace HostFilter and RoleFilter with Filter in Configuration
Create list of filters from command arguments and ENV vars
Intercept the existing on() method to apply external filters
Update roles_for() and fetch_primary() to use internal filters
This commit is contained in:
Nick Townsend 2014-09-14 11:53:02 -07:00
parent c55ae63044
commit 5bae7fb40c
16 changed files with 323 additions and 442 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
CHANGELOG.md
View file

@ -6,6 +6,14 @@ Reverse Chronological Order:
https://github.com/capistrano/capistrano/compare/v3.2.1...HEAD
* Enhancements (@townsen)
* _External_ Host and Role filtering now affects only `on()` commands
and not the `roles()`, `release_roles()` and `primary()` methods.
* _Internal_ Host and Role filtering affects the `roles()`, `release_roles()`
and `primary()` methods.
* Host and Role filtering now supports Regular expressions
* See the README.md file for a comprehensive discussion of these changes
* Pushing again to trigger another build (I have a seemingly random build fail) (@townsen)
* Enhancements (@townsen)
* Added set_if_empty method to DSL to allow conditional setting

58
README.md
View file

@ -267,6 +267,64 @@ __Support removed__ for following variables:
|:---------------------:|---------------------------------------------------------------------|-----------------------------------------------------------------|
| `:copy_exclude` | The (optional) array of files and/or folders excluded from deploy | Replaced by Git's native `.gitattributes`, see [#515](https://github.com/capistrano/capistrano/issues/515) for more info. |
## Host and Role Filtering
Capistrano enables the declaration of servers and roles, each of which may have properties
associated with them. Tasks are then able to use these definitions in two distinct ways:
* To execute commands on remote hosts: using the `on()` method (provided by SSHKit), and
* To determine configurations: typically by using the `roles()` method (and relatives) outside the scope of `on()`
An example of the latter would be to create a list of available web servers in order to
automate the setup of an F5 pool.
A problem with this arises when _filters_ are used. Filters are designed to limit the
actual set of hosts that are used to a subset of those in the overall stage, but how
should that apply to the above?
If the filter applies to both the _command_ and _configuration_ aspects, any configuration
files deployed will not be the same as those on the hosts excluded by the filters. On the
other hand if the filter applies only to the _command_ aspect, then any configuration
files deployed will be identical across the stage.
Consider also the different ways in which filters may be specified. Externally:
* Via environment variables HOSTS and ROLES
* Via command line options `--hosts` and `--roles`
And internally:
* Via the `:filter` variable
* Via options passed to the `roles()` method (and implicitly in methods like `release_roles()`)
Currently, when a filter is applied via __any__ of the current methods, it
affects __both__ the _command_ and the _configuration_ aspects.
For practical uses this behaviour needs refining. A core principle of Capistrano is that
the stage file is the complete embodiment of the configuration (possibly including
settings from deploy.rb and Capfile), and therefore any filtering of the configuration
should be declared there. Put the other way, an external filter, done for the purposes of
limiting which hosts commands are executed on, should not affect the overall
configuration.
So this fix makes the external filters only apply to commands issued: ie. they restrict
the hosts that an `on()` method will use, the will not affect the `roles()` method. On the
other hand internal filters will always apply
By making this distinction two distinct usage models can be catered for. When a subset of
servers and/or roles are deployed to:
* With a configuration that reflects only that subset. This can be achieved by
using the `set :filter, hosts: [a,b], roles: [:web,:app]` approach.
* With a configuration that reflects the entire stage. This is useful when deploying a new
server to an existing configuration. This can be achieved by using either of the
external filtering methods.
The former corresponds to the existing behaviour, although done slightly differently.
The latter is new behaviour which is a more common use case.
We also change external filters so that they can use regular expressions. If either
a host or role name in a filter doesn't match `/^[-\w.]*$/` then it's assumed to be
a regular expression.
## SSHKit
[SSHKit](https://github.com/leehambley/sshkit) is the driver for SSH

7
lib/capistrano/all.rb
View file

@ -1,6 +1,13 @@
require 'rake'
require 'sshkit'
require 'sshkit/dsl'
module SSHKit
module DSL
alias_method :sshkit_on, :on
end
end
require 'io/console'
Rake.application.options.trace = true

15
lib/capistrano/application.rb
View file

@ -21,7 +21,7 @@ module Capistrano
switch =~ /--#{Regexp.union(not_applicable_to_capistrano)}/
end
super.push(version, roles, dry_run, hostfilter)
super.push(version, dry_run, roles, hostfilter)
end
def handle_options
@ -40,6 +40,11 @@ module Capistrano
opts.separator "Invoke (or simulate invoking) a task:"
opts.separator " bundle exec cap [--dry-run] STAGE TASK"
opts.separator ""
opts.separator "Host and Role Filters:"
opts.separator " Host and role patterns may be specified as a comma separated list"
opts.separator " each item of which is treated either as a literal match (if it contains"
opts.separator " just the characters A-Za-z0-9-_.) or a regular expression (otherwise)."
opts.separator ""
opts.separator "Advanced options:"
opts.on_tail("-h", "--help", "-H", "Display this help message.") do
@ -107,18 +112,18 @@ module Capistrano
def roles
['--roles ROLES', '-r',
"Filter command to only apply to these roles (separate multiple roles with a comma)",
"Run SSH commands only on hosts matching these roles (see syntax above)",
lambda { |value|
Configuration.env.set(:filter, roles: value.split(","))
Configuration.env.add_external_filter(:role, value.split(","))
}
]
end
def hostfilter
['--hosts HOSTS', '-z',
"Filter command to only apply to these hosts (separate multiple hosts with a comma)",
"Run SSH commands only on matching hosts (see syntax above)",
lambda { |value|
Configuration.env.set(:filter, hosts: value.split(","))
Configuration.env.add_external_filter(:host, value.split(","))
}
]
end

15
lib/capistrano/configuration.rb
View file

@ -1,6 +1,7 @@
require_relative 'configuration/filter'
require_relative 'configuration/question'
require_relative 'configuration/servers'
require_relative 'configuration/server'
require_relative 'configuration/servers'
module Capistrano
class Configuration
@ -86,6 +87,14 @@ module Capistrano
@timestamp ||= Time.now.utc
end
def add_external_filter(type, values)
external_filters << Filter.new(type, values)
end
def filter list
external_filters.reduce(list){|l,f| f.filter list}
end
private
def servers
@ -96,6 +105,10 @@ module Capistrano
@config ||= Hash.new
end
def external_filters
@external_filters ||= []
end
def fetch_for(key, default, &block)
if block_given?
config.fetch(key, &block)

39
lib/capistrano/configuration/filter.rb Normal file
View file

@ -0,0 +1,39 @@
require 'capistrano/configuration'
module Capistrano
class Configuration
class Filter
def initialize type, values = nil
raise "Invalid filter type #{type}" unless [:host,:role].include? type
av = Array(values)
@type = type
@mode = case
when av.size == 0 then :none
when av.include?(:all) then :all
else
Regexp.union av.map { |v|
case v
when Regexp then v
else
vs = v.to_s
vs =~ /^[-\w.]*$/ ? vs : Regexp.new(vs)
end
}
end
end
def filter servers
case @mode
when :none then return []
when :all then return servers
else
case @type
when :host
servers.select {|s| @mode.match s.hostname}
when :role
servers.select {|s| s.roles.any? {|r| @mode.match r} }
end
end
end
end
end
end

2
lib/capistrano/configuration/server.rb
View file

@ -11,11 +11,13 @@ module Capistrano
def add_roles(roles)
Array(roles).each { |role| add_role(role) }
self
end
alias roles= add_roles
def add_role(role)
roles.add role.to_sym
self
end
def has_role?(role)

35
lib/capistrano/configuration/servers.rb
View file

@ -1,6 +1,7 @@
require 'set'
require_relative 'servers/role_filter'
require_relative 'servers/host_filter'
require 'capistrano/configuration'
require 'capistrano/configuration/filter'
module Capistrano
class Configuration
class Servers
@ -17,11 +18,16 @@ module Capistrano
def roles_for(names)
options = extract_options(names)
fetch_roles(names, options)
fia = Array(Filter.new(:role, names))
fs = Configuration.env.fetch(:filter,{})
fia << Filter.new(:host, fs[:host]) if fs[:host]
fia << Filter.new(:role, fs[:role]) if fs[:role]
s = fia.reduce(servers){|m,o| o.filter m}
s.select { |server| server.select?(options) }
end
def fetch_primary(role)
hosts = fetch(role)
hosts = roles_for([role])
hosts.find(&:primary) || hosts.first
end
@ -35,27 +41,6 @@ module Capistrano
servers.find { |server| server.matches? Server[host] } || Server[host]
end
def fetch(role)
servers.find_all { |server| server.has_role? role}
end
def fetch_roles(required, options)
filter_roles = RoleFilter.for(required, available_roles)
HostFilter.for(select(servers_with_roles(filter_roles), options))
end
def servers_with_roles(roles)
roles.flat_map { |role| fetch role }.uniq
end
def select(servers, options)
servers.select { |server| server.select?(options) }
end
def available_roles
servers.flat_map { |server| server.roles_array }.uniq
end
def servers
@servers ||= Set.new
end

82
lib/capistrano/configuration/servers/host_filter.rb
View file

@ -1,82 +0,0 @@
module Capistrano
class Configuration
class Servers
class HostFilter
def initialize(available)
@available = available
end
def self.for(available)
new(available).hosts
end
def hosts
if host_filter.any?
@available.select { |server| host_filter.include? server.hostname }
else
@available
end
end
private
def filter
if host_filter.any?
host_filter
else
@available
end
end
def host_filter
env_filter | configuration_filter
end
def configuration_filter
ConfigurationFilter.new.hosts
end
def env_filter
EnvFilter.new.hosts
end
class ConfigurationFilter
def hosts
if filter
Array(filter.fetch(:hosts, []))
else
[]
end
end
def config
Configuration.env
end
def filter
config.fetch(:filter) || config.fetch(:select)
end
end
class EnvFilter
def hosts
if filter
filter.split(',')
else
[]
end
end
def filter
ENV['HOSTS']
end
end
end
end
end
end

86
lib/capistrano/configuration/servers/role_filter.rb
View file

@ -1,86 +0,0 @@
module Capistrano
class Configuration
class Servers
class RoleFilter
def initialize(required, available)
@required, @available = required, available
end
def self.for(required, available)
new(required, available).roles
end
def roles
if required.include?(:all)
available
else
required.select { |name| available.include? name }
end
end
private
def required
Array(@required).flat_map(&:to_sym)
end
def available
if role_filter.any?
role_filter
else
@available
end
end
def role_filter
env_filter | configuration_filter
end
def configuration_filter
ConfigurationFilter.new.roles
end
def env_filter
EnvFilter.new.roles
end
class ConfigurationFilter
def roles
if filter
Array(filter.fetch(:roles, [])).map(&:to_sym)
else
[]
end
end
def config
Configuration.env
end
def filter
config.fetch(:filter) || config.fetch(:select)
end
end
class EnvFilter
def roles
if filter
filter.split(',').map(&:to_sym)
else
[]
end
end
def filter
ENV['ROLES']
end
end
end
end
end
end

6
lib/capistrano/dsl.rb
View file

@ -48,6 +48,12 @@ module Capistrano
def lock(locked_version)
VersionValidator.new(locked_version).verify
end
# Having intercepted the SSHKit on() method we can filter externally
def on(hosts, options={}, &block)
sshkit_on(Configuration.env.filter(hosts), options, &block)
end
end
end
self.extend Capistrano::DSL

31
spec/integration/dsl_spec.rb
View file

@ -100,6 +100,37 @@ describe Capistrano::DSL do
end
end
describe 'setting an internal host filter' do
subject { dsl.roles(:app) }
it 'returns one' do
dsl.set :filter, { host: 'example3.com' }
expect(subject.map(&:hostname)).to eq(['example3.com'])
end
end
describe 'setting an internal role filter' do
subject { dsl.roles(:app) }
it 'returns one' do
dsl.set :filter, { role: :web }
expect(subject.map(&:hostname)).to eq(['example3.com'])
end
end
describe 'setting an internal host and role filter' do
subject { dsl.roles(:app) }
it 'returns one' do
dsl.set :filter, { role: :web, host: 'example1.com' }
expect(subject.map(&:hostname)).to be_empty
end
end
describe 'setting an internal regexp host filter' do
subject { dsl.roles(:all) }
it 'works' do
dsl.set :filter, { host: /1/ }
expect(subject.map(&:hostname)).to eq(['example1.com'])
end
end
end
describe 'when defining role with reserved name' do

75
spec/lib/capistrano/configuration/filter_spec.rb Normal file
View file

@ -0,0 +1,75 @@
require 'spec_helper'
module Capistrano
class Configuration
describe Filter do
let(:available) { [ Server.new('server1').add_roles([:web,:db]),
Server.new('server2').add_role(:web),
Server.new('server3').add_role(:redis),
Server.new('server4').add_role(:db) ] }
describe '#new' do
it "won't create an invalid type of filter" do
expect {
f = Filter.new(:zarg)
}.to raise_error RuntimeError
end
it 'creates an empty host filter' do
expect(Filter.new(:host).filter(available)).to be_empty
end
it 'creates a null host filter' do
expect(Filter.new(:host, :all).filter(available)).to eq(available)
end
it 'creates an empty role filter' do
expect(Filter.new(:role).filter(available)).to be_empty
end
it 'creates a null role filter' do
expect(Filter.new(:role, :all).filter(available)).to eq(available)
end
end
describe 'host filter' do
it 'returns all hosts matching string' do
set = Filter.new(:host, %w{server1 server3}).filter(available)
expect(set.map(&:hostname)).to eq(%w{server1 server3})
end
it 'returns all hosts matching regexp' do
set = Filter.new(:host, 'server[1,3]$').filter(available)
expect(set.map(&:hostname)).to eq(%w{server1 server3})
end
end
describe 'role filter' do
it 'returns all hosts' do
set = Filter.new(:role, [:all]).filter(available)
expect(set.size).to eq(available.size)
expect(set.first.hostname).to eq('server1')
end
it 'returns hosts in a single role' do
set = Filter.new(:role, [:web]).filter(available)
expect(set.size).to eq(2)
expect(set.map(&:hostname)).to eq(%w{ server1 server2 })
end
it 'returns hosts in multiple roles' do
set = Filter.new(:role, [:web, :db]).filter(available)
expect(set.size).to eq(3)
expect(set.map(&:hostname)).to eq(%w{ server1 server2 server4 })
end
it 'returns hosts with regex role selection' do
set = Filter.new(:role, '^red').filter(available)
expect(set.map(&:hostname)).to eq(%w{ server3 })
end
it 'returns hosts with regex role selection' do
set = Filter.new(:role, /red/).filter(available)
expect(set.map(&:hostname)).to eq(%w{ server3 })
end
end
end
end
end

84
spec/lib/capistrano/configuration/servers/host_filter_spec.rb
View file

@ -1,84 +0,0 @@
require 'spec_helper'
module Capistrano
class Configuration
class Servers
describe HostFilter do
let(:host_filter) { HostFilter.new(available) }
let(:available) { [ Server.new('server1'), Server.new('server2'), Server.new('server3') ] }
describe '#new' do
it 'takes one array of hostnames' do
expect(host_filter)
end
end
describe '.for' do
subject { HostFilter.for(available) }
context 'without env vars' do
it 'returns all available hosts' do
expect(subject).to eq available
end
end
context 'with ENV vars' do
before do
ENV.stubs(:[]).with('HOSTS').returns('server1,server2')
end
it 'returns all required hosts defined in HOSTS' do
expect(subject).to eq [Server.new('server1'), Server.new('server2')]
end
end
context 'with configuration filters' do
before do
Configuration.env.set(:filter, hosts: %w{server1 server2})
end
it 'returns all required hosts defined in the filter' do
expect(subject).to eq [Server.new('server1'), Server.new('server2')]
end
after do
Configuration.env.delete(:filter)
end
end
context 'with a single configuration filter' do
before do
Configuration.env.set(:filter, hosts: 'server3')
end
it 'returns all required hosts defined in the filter' do
expect(subject).to eq [Server.new('server3')]
end
after do
Configuration.env.delete(:filter)
end
end
context 'with configuration filters and ENV vars' do
before do
Configuration.env.set(:filter, hosts: 'server1')
ENV.stubs(:[]).with('HOSTS').returns('server3')
end
it 'returns all required hosts defined in the filter' do
expect(subject).to eq [Server.new('server1'), Server.new('server3')]
end
after do
Configuration.env.delete(:filter)
end
end
end
end
end
end
end

140
spec/lib/capistrano/configuration/servers/role_filter_spec.rb
View file

@ -1,140 +0,0 @@
require 'spec_helper'
module Capistrano
class Configuration
class Servers
describe RoleFilter do
let(:role_filter) { RoleFilter.new(required, available) }
let(:required) { [] }
let(:available) { [:web, :app, :db] }
describe '#new' do
it 'takes two arrays of role names' do
expect(role_filter)
end
end
describe '.for' do
subject { RoleFilter.for(required, available) }
context 'without env vars' do
context ':all required' do
let(:required) { [:all] }
it 'returns all available names' do
expect(subject).to eq available
end
end
context 'role names required' do
let(:required) { [:web, :app] }
it 'returns all required names' do
expect(subject).to eq required
end
end
end
context 'with ENV vars' do
before do
ENV.stubs(:[]).with('ROLES').returns('app,web')
end
context ':all required' do
let(:required) { [:all] }
it 'returns available names defined in ROLES' do
expect(subject).to eq [:app, :web]
end
end
context 'role names required' do
let(:required) { [:web, :db] }
it 'returns all required names defined in ROLES' do
expect(subject).to eq [:web]
end
end
end
context 'with configuration filters' do
before do
Configuration.env.set(:filter, roles: %w{app web})
end
context ':all required' do
let(:required) { [:all] }
it 'returns available names defined in the filter' do
expect(subject).to eq [:app, :web]
end
end
context 'role names required' do
let(:required) { [:web, :db] }
it 'returns all required names defined in the filter' do
expect(subject).to eq [:web]
end
end
after do
Configuration.env.delete(:filter)
end
end
context 'with a single configuration filter' do
before do
Configuration.env.set(:filter, roles: 'web')
end
context ':all required' do
let(:required) { [:all] }
it 'returns available names defined in the filter' do
expect(subject).to eq [:web]
end
end
context 'role names required' do
let(:required) { [:web, :db] }
it 'returns all required names defined in the filter' do
expect(subject).to eq [:web]
end
end
after do
Configuration.env.delete(:filter)
end
end
context 'with configuration filters and ENV vars' do
before do
Configuration.env.set(:filter, roles: %w{app})
ENV.stubs(:[]).with('ROLES').returns('web')
end
context ':all required' do
let(:required) { [:all] }
it 'returns available names defined in the filter' do
expect(subject).to eq [:web, :app]
end
end
context 'role names required' do
let(:required) { [:web, :db] }
it 'returns all required names defined in the filter' do
expect(subject).to eq [:web]
end
end
after do
Configuration.env.delete(:filter)
end
end
end
end
end
end
end

82
spec/lib/capistrano/configuration/servers_spec.rb
View file

@ -68,6 +68,9 @@ module Capistrano
end
describe 'finding the primary server' do
after do
Configuration.reset!
end
it 'takes the first server if none have the primary property' do
servers.add_role(:app, %w{1 2})
expect(servers.fetch_primary(:app).hostname).to eq('1')
@ -78,6 +81,13 @@ module Capistrano
servers.add_host('2', primary: true)
expect(servers.fetch_primary(:app).hostname).to eq('2')
end
it 'honours any internal filters' do
Configuration.env.set :filter, { host: '1'}
servers.add_role(:app, %w{1 2})
servers.add_host('2', primary: true)
expect(servers.fetch_primary(:app).hostname).to eq('1')
end
end
describe 'fetching servers' do
@ -218,11 +228,9 @@ module Capistrano
end
describe 'filtering roles' do
describe 'filtering roles internally' do
before do
ENV.stubs(:[]).with('ROLES').returns('web,db')
ENV.stubs(:[]).with('HOSTS').returns(nil)
servers.add_host('1', roles: :app, active: true)
servers.add_host('2', roles: :app)
servers.add_host('3', roles: :web)
@ -232,31 +240,67 @@ module Capistrano
subject { servers.roles_for(roles).map(&:hostname) }
context 'when selecting all roles' do
let(:roles) { [:all] }
context 'with the ROLES environment variable set' do
it 'returns the roles specified by ROLE' do
expect(subject).to eq %w{3 4 5}
before do
ENV.stubs(:[]).with('ROLES').returns('web,db')
ENV.stubs(:[]).with('HOSTS').returns(nil)
end
context 'when selecting all roles' do
let(:roles) { [:all] }
it 'ignores it' do
expect(subject).to eq %w{1 2 3 4 5}
end
end
context 'when selecting specific roles' do
let(:roles) { [:app, :web] }
it 'ignores it' do
expect(subject).to eq %w{1 2 3 4}
end
end
context 'when selecting roles not included in ROLE' do
let(:roles) { [:app] }
it 'ignores it' do
expect(subject).to eq %w{1 2}
end
end
end
context 'when selecting roles included in ROLE' do
let(:roles) { [:app, :web] }
context 'with the HOSTS environment variable set' do
it 'returns only roles that match ROLE' do
expect(subject).to eq %w{3 4}
before do
ENV.stubs(:[]).with('ROLES').returns(nil)
ENV.stubs(:[]).with('HOSTS').returns('3,5')
end
context 'when selecting all roles' do
let(:roles) { [:all] }
it 'ignores it' do
expect(subject).to eq %w{1 2 3 4 5}
end
end
context 'when selecting specific roles' do
let(:roles) { [:app, :web] }
it 'ignores it' do
expect(subject).to eq %w{1 2 3 4}
end
end
context 'when selecting no roles' do
let(:roles) { [] }
it 'ignores it' do
expect(subject).to be_empty
end
end
end
context 'when selecting roles not included in ROLE' do
let(:roles) { [:app] }
it 'is empty' do
expect(subject).to be_empty
end
end
end
end
end
end