kotovalexarian-likes-gitlab/gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity

Merge branch 'caches' into 'master'

Browse source 
Allow to define cache in `.gitlab-ci.yml`

This extends `.gitlab-ci.yml` syntax to allow specifying caching files and directories between builds, making it easy to preserve ex. gems.

```
cache:
  paths:
  - .bundle
  - vendor/

before_script:
- bundle install --path vendor/

rspec:
  script:
  - bundle exec rspec
```

This is based on Build Artifacts changes.

/cc@dzaporozhets 

See merge request !1786
This commit is contained in:
Kamil Trzciński 2015-11-12 09:39:56 +00:00
commit 9573cf6cc7
4 changed files with 175 additions and 4 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
CHANGELOG
View file

@ -20,6 +20,7 @@ v 8.2.0 (unreleased)
- Send build name and stage in CI notification e-mail
- Extend yml syntax for only and except to support specifying repository path
- Enable shared runners to all new projects
- Allow to define cache in `.gitlab-ci.yml`
- Fix: 500 error returned if destroy request without HTTP referer (Kazuki Shimizu)
- Remove deprecated CI events from project settings page
- [API] Add ability to fetch the commit ID of the last commit that actually touched a file

66
doc/ci/yaml/README.md
View file

@ -56,6 +56,7 @@ There are a few `keywords` that can't be used as job names:
| types | optional | Alias for `stages` |
| before_script | optional | Define commands prepended for each job's script |
| variables | optional | Define build variables |
| cache | optional | Define list of files that should be cached between subsequent runs |
### image and services
This allows to specify a custom Docker image and a list of services that can be used for time of the build.
@ -110,6 +111,19 @@ These variables can be later used in all executed commands and scripts.
The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them.
### cache
`cache` is used to specify list of files and directories which should be cached between builds.
**The global setting allows to specify default cached files for all jobs.**
To cache all git untracked files and files in `binaries`:
```
cache:
untracked: true
paths:
- binaries/
```
## Jobs
`.gitlab-ci.yml` allows you to specify an unlimited number of jobs.
Each job has to have a unique `job_name`, which is not one of the keywords mentioned above.
@ -142,6 +156,7 @@ job_name:
| allow_failure | optional | Allow build to fail. Failed build doesn't contribute to commit status |
| when | optional | Define when to run build. Can be `on_success`, `on_failure` or `always` |
| artifacts | optional | Define list build artifacts |
| cache | optional | Define list of files that should be cached between subsequent runs |
### script
`script` is a shell script which is executed by runner. The shell script is prepended with `before_script`.
@ -288,6 +303,57 @@ The artifacts will be send after the build success to GitLab and will be accessi
This feature requires GitLab Runner v0.7.0 or higher.
### cache
`cache` is used to specify list of files and directories which should be cached between builds.
1. Cache all files in `binaries` and `.config`:
```
rspec:
script: test
cache:
paths:
- binaries/
- .config
```
2. Cache all git untracked files:
```
rspec:
script: test
cache:
untracked: true
```
3. Cache all git untracked files and files in `binaries`:
```
rspec:
script: test
cache:
untracked: true
paths:
- binaries/
```
4. Locally defined cache overwrites globally defined options. This will cache only `binaries/`:
```
cache:
paths:
- my/files
rspec:
script: test
cache:
paths:
- binaries/
```
The cache is provided on best effort basis, so don't expect that cache will be present.
For implementation details please check GitLab Runner.
This feature requires GitLab Runner v0.7.0 or higher.
## Validate the .gitlab-ci.yml
Each instance of GitLab CI has an embedded debug tool called Lint.
You can find the link to the Lint in the project's settings page or use short url `/lint`.

30
lib/ci/gitlab_ci_yaml_processor.rb
View file

@ -4,10 +4,10 @@ module Ci
DEFAULT_STAGES = %w(build test deploy)
DEFAULT_STAGE = 'test'
ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables]
ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services, :allow_failure, :type, :stage, :when, :artifacts]
ALLOWED_YAML_KEYS = [:before_script, :image, :services, :types, :stages, :variables, :cache]
ALLOWED_JOB_KEYS = [:tags, :script, :only, :except, :type, :image, :services, :allow_failure, :type, :stage, :when, :artifacts, :cache]
attr_reader :before_script, :image, :services, :variables, :path
attr_reader :before_script, :image, :services, :variables, :path, :cache
def initialize(config, path = nil)
@config = YAML.load(config)
@ -46,6 +46,7 @@ module Ci
@services = @config[:services]
@stages = @config[:stages] || @config[:types]
@variables = @config[:variables] || {}
@cache = @config[:cache]
@config.except!(*ALLOWED_YAML_KEYS)
# anything that doesn't have script is considered as unknown
@ -78,7 +79,8 @@ module Ci
options: {
image: job[:image] || @image,
services: job[:services] || @services,
artifacts: job[:artifacts]
artifacts: job[:artifacts],
cache: job[:cache] || @cache,
}.compact
}
end
@ -112,6 +114,16 @@ module Ci
raise ValidationError, "variables should be a map of key-valued strings"
end
if @cache
if @cache[:untracked] && !validate_boolean(@cache[:untracked])
raise ValidationError, "cache:untracked parameter should be an boolean"
end
if @cache[:paths] && !validate_array_of_strings(@cache[:paths])
raise ValidationError, "cache:paths parameter should be an array of strings"
end
end
@jobs.each do |name, job|
validate_job!(name, job)
end
@ -160,6 +172,16 @@ module Ci
raise ValidationError, "#{name} job: except parameter should be an array of strings"
end
if job[:cache]
if job[:cache][:untracked] && !validate_boolean(job[:cache][:untracked])
raise ValidationError, "#{name} job: cache:untracked parameter should be an boolean"
end
if job[:cache][:paths] && !validate_array_of_strings(job[:cache][:paths])
raise ValidationError, "#{name} job: cache:paths parameter should be an array of strings"
end
end
if job[:artifacts]
if job[:artifacts][:untracked] && !validate_boolean(job[:artifacts][:untracked])
raise ValidationError, "#{name} job: artifacts:untracked parameter should be an boolean"

82
spec/lib/ci/gitlab_ci_yaml_processor_spec.rb
View file

@ -333,6 +333,60 @@ module Ci
end
end
describe "Caches" do
it "returns cache when defined globally" do
config = YAML.dump({
cache: { paths: ["logs/", "binaries/"], untracked: true },
rspec: {
script: "rspec"
}
})
config_processor = GitlabCiYamlProcessor.new(config)
expect(config_processor.builds_for_stage_and_ref("test", "master").size).to eq(1)
expect(config_processor.builds_for_stage_and_ref("test", "master").first[:options][:cache]).to eq(
paths: ["logs/", "binaries/"],
untracked: true,
)
end
it "returns cache when defined in a job" do
config = YAML.dump({
rspec: {
cache: { paths: ["logs/", "binaries/"], untracked: true },
script: "rspec"
}
})
config_processor = GitlabCiYamlProcessor.new(config)
expect(config_processor.builds_for_stage_and_ref("test", "master").size).to eq(1)
expect(config_processor.builds_for_stage_and_ref("test", "master").first[:options][:cache]).to eq(
paths: ["logs/", "binaries/"],
untracked: true,
)
end
it "overwrite cache when defined for a job and globally" do
config = YAML.dump({
cache: { paths: ["logs/", "binaries/"], untracked: true },
rspec: {
script: "rspec",
cache: { paths: ["test/"], untracked: false },
}
})
config_processor = GitlabCiYamlProcessor.new(config)
expect(config_processor.builds_for_stage_and_ref("test", "master").size).to eq(1)
expect(config_processor.builds_for_stage_and_ref("test", "master").first[:options][:cache]).to eq(
paths: ["test/"],
untracked: false,
)
end
end
describe "Artifacts" do
it "returns artifacts when defined" do
config = YAML.dump({
@ -542,6 +596,34 @@ module Ci
GitlabCiYamlProcessor.new(config)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: artifacts:paths parameter should be an array of strings")
end
it "returns errors if cache:untracked is not an array of strings" do
config = YAML.dump({ cache: { untracked: "string" }, rspec: { script: "test" } })
expect do
GitlabCiYamlProcessor.new(config)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "cache:untracked parameter should be an boolean")
end
it "returns errors if cache:paths is not an array of strings" do
config = YAML.dump({ cache: { paths: "string" }, rspec: { script: "test" } })
expect do
GitlabCiYamlProcessor.new(config)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "cache:paths parameter should be an array of strings")
end
it "returns errors if job cache:untracked is not an array of strings" do
config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", cache: { untracked: "string" } } })
expect do
GitlabCiYamlProcessor.new(config)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: cache:untracked parameter should be an boolean")
end
it "returns errors if job cache:paths is not an array of strings" do
config = YAML.dump({ types: ["build", "test"], rspec: { script: "test", cache: { paths: "string" } } })
expect do
GitlabCiYamlProcessor.new(config)
end.to raise_error(GitlabCiYamlProcessor::ValidationError, "rspec job: cache:paths parameter should be an array of strings")
end
end
end
end