gitlab-org--gitlab-foss/app/models/ci/pipeline.rb

397 lines
9.8 KiB
Ruby
Raw Normal View History

2015-08-25 21:42:46 -04:00
module Ci
class Pipeline < ActiveRecord::Base
2015-08-25 21:42:46 -04:00
extend Ci::Model
2016-08-24 22:55:32 -04:00
include HasStatus
include Importable
include AfterCommitQueue
2015-09-24 11:09:33 -04:00
2017-03-17 19:06:11 -04:00
belongs_to :project
2016-07-15 09:42:29 -04:00
belongs_to :user
2016-06-03 07:58:35 -04:00
has_many :statuses, class_name: 'CommitStatus', foreign_key: :commit_id
has_many :builds, foreign_key: :commit_id
has_many :trigger_requests, dependent: :destroy, foreign_key: :commit_id
2015-08-25 21:42:46 -04:00
2017-02-22 17:35:08 -05:00
delegate :id, to: :project, prefix: true
2017-02-21 19:40:04 -05:00
validates :sha, presence: { unless: :importing? }
validates :ref, presence: { unless: :importing? }
validates :status, presence: { unless: :importing? }
validate :valid_commit_sha, unless: :importing?
2015-08-25 21:42:46 -04:00
after_create :keep_around_commits, unless: :importing?
after_create :refresh_build_status_cache
2016-04-11 10:55:40 -04:00
state_machine :status, initial: :created do
2016-08-12 07:57:58 -04:00
event :enqueue do
2016-08-12 06:23:47 -04:00
transition created: :pending
transition [:success, :failed, :canceled, :skipped] => :running
end
event :run do
2016-10-20 03:33:44 -04:00
transition any - [:running] => :running
end
event :skip do
2016-10-20 03:33:44 -04:00
transition any - [:skipped] => :skipped
end
event :drop do
2016-10-20 03:33:44 -04:00
transition any - [:failed] => :failed
end
event :succeed do
2016-10-20 03:33:44 -04:00
transition any - [:success] => :success
end
event :cancel do
2016-10-20 03:33:44 -04:00
transition any - [:canceled] => :canceled
end
event :block do
transition any - [:manual] => :manual
end
2016-10-14 13:08:48 -04:00
# IMPORTANT
# Do not add any operations to this state_machine
# Create a separate worker for each new operation
before_transition [:created, :pending] => :running do |pipeline|
pipeline.started_at = Time.now
end
before_transition any => [:success, :failed, :canceled] do |pipeline|
pipeline.finished_at = Time.now
pipeline.update_duration
end
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
after_transition [:created, :pending] => :running do |pipeline|
2016-10-14 13:08:48 -04:00
pipeline.run_after_commit { PipelineMetricsWorker.perform_async(id) }
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
end
after_transition any => [:success] do |pipeline|
2016-10-14 13:08:48 -04:00
pipeline.run_after_commit { PipelineMetricsWorker.perform_async(id) }
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
end
after_transition [:created, :pending, :running] => :success do |pipeline|
pipeline.run_after_commit { PipelineSuccessWorker.perform_async(id) }
end
after_transition do |pipeline, transition|
2016-10-12 07:44:33 -04:00
next if transition.loopback?
pipeline.run_after_commit do
PipelineHooksWorker.perform_async(id)
end
end
after_transition any => [:success, :failed] do |pipeline|
pipeline.run_after_commit do
PipelineNotificationWorker.perform_async(pipeline.id)
end
end
end
# ref can't be HEAD or SHA, can only be branch/tag name
scope :latest, ->(ref = nil) do
max_id = unscope(:select)
.select("max(#{quoted_table_name}.id)")
.group(:ref, :sha)
if ref
where(ref: ref, id: max_id.where(ref: ref))
else
where(id: max_id)
end
end
def self.latest_status(ref = nil)
latest(ref).status
end
def self.latest_successful_for(ref)
success.latest(ref).order(id: :desc).first
end
def self.latest_successful_for_refs(refs)
success.latest(refs).order(id: :desc).each_with_object({}) do |pipeline, hash|
hash[pipeline.ref] ||= pipeline
end
end
2015-08-25 21:42:46 -04:00
def self.truncate_sha(sha)
sha[0...8]
end
def self.total_duration
2016-08-15 06:50:02 -04:00
where.not(duration: nil).sum(:duration)
end
2016-12-20 05:00:56 -05:00
def stage(name)
stage = Ci::Stage.new(self, name: name)
stage unless stage.statuses_count.zero?
end
def stages_count
statuses.select(:stage).distinct.count
end
2016-12-06 08:49:37 -05:00
def stages_name
statuses.order(:stage_idx).distinct.
pluck(:stage, :stage_idx).map(&:first)
2016-12-06 08:49:37 -05:00
end
def stages
# TODO, this needs refactoring, see gitlab-ce#26481.
stages_query = statuses
.group('stage').select(:stage).order('max(stage_idx)')
status_sql = statuses.latest.where('stage=sg.stage').status_sql
warnings_sql = statuses.latest.select('COUNT(*)')
.where('stage=sg.stage').failed_but_allowed.to_sql
stages_with_statuses = CommitStatus.from(stages_query, :sg)
.pluck('sg.stage', status_sql, "(#{warnings_sql})")
stages_with_statuses.map do |stage|
Ci::Stage.new(self, Hash[%i[name status warnings].zip(stage)])
end
end
def artifacts
2016-12-15 16:06:39 -05:00
builds.latest.with_artifacts_not_expired.includes(project: [:namespace])
end
2015-08-25 21:42:46 -04:00
def valid_commit_sha
2015-12-11 11:57:04 -05:00
if self.sha == Gitlab::Git::BLANK_SHA
2015-08-25 21:42:46 -04:00
self.errors.add(:sha, " cant be 00000000 (branch removal)")
end
end
def git_author_name
commit.try(:author_name)
2015-08-25 21:42:46 -04:00
end
def git_author_email
commit.try(:author_email)
2015-08-25 21:42:46 -04:00
end
def git_commit_message
commit.try(:message)
2015-08-25 21:42:46 -04:00
end
def git_commit_title
commit.try(:title)
end
2015-08-25 21:42:46 -04:00
def short_sha
Ci::Pipeline.truncate_sha(sha)
2015-08-25 21:42:46 -04:00
end
def commit
2015-12-04 06:55:23 -05:00
@commit ||= project.commit(sha)
2015-08-25 21:42:46 -04:00
rescue
nil
end
2016-03-31 13:51:28 -04:00
def branch?
!tag?
end
def manual_actions
2016-12-15 16:06:39 -05:00
builds.latest.manual_actions.includes(project: [:namespace])
end
def stuck?
builds.pending.includes(:project).any?(&:stuck?)
2016-07-16 12:39:58 -04:00
end
2016-03-31 13:51:28 -04:00
def retryable?
builds.latest.failed_or_canceled.any?(&:retryable?)
2016-03-31 13:51:28 -04:00
end
def cancelable?
statuses.cancelable.any?
end
2016-05-09 19:26:13 -04:00
def cancel_running
Gitlab::OptimisticLocking.retry_lock(
statuses.cancelable) do |cancelable|
cancelable.find_each(&:cancel)
end
2016-05-09 19:26:13 -04:00
end
def retry_failed(current_user)
Ci::RetryPipelineService.new(project, current_user)
.execute(self)
2016-05-09 19:26:13 -04:00
end
def mark_as_processable_after_stage(stage_idx)
builds.skipped.after_stage(stage_idx).find_each(&:process)
end
2016-04-13 07:01:08 -04:00
def latest?
return false unless ref
commit = project.commit(ref)
return false unless commit
commit.sha == sha
end
2016-04-12 13:57:22 -04:00
def triggered?
trigger_requests.any?
end
2015-10-06 06:01:16 -04:00
def retried
@retried ||= (statuses.order(id: :desc) - statuses.latest)
2015-08-25 21:42:46 -04:00
end
def coverage
2016-04-13 14:51:03 -04:00
coverage_array = statuses.latest.map(&:coverage).compact
2015-12-04 06:55:23 -05:00
if coverage_array.size >= 1
'%.2f' % (coverage_array.reduce(:+) / coverage_array.size)
2015-08-25 21:42:46 -04:00
end
end
def config_builds_attributes
return [] unless config_processor
config_processor.
builds_for_ref(ref, tag?, trigger_requests.first).
sort_by { |build| build[:stage_idx] }
end
2016-07-14 10:58:05 -04:00
def has_warnings?
builds.latest.failed_but_allowed.any?
end
2015-08-25 21:42:46 -04:00
def config_processor
return nil unless ci_yaml_file
return @config_processor if defined?(@config_processor)
@config_processor ||= begin
Ci::GitlabCiYamlProcessor.new(ci_yaml_file, project.path_with_namespace)
rescue Ci::GitlabCiYamlProcessor::ValidationError, Psych::SyntaxError => e
self.yaml_errors = e.message
nil
rescue
self.yaml_errors = 'Undefined error'
nil
end
2015-08-25 21:42:46 -04:00
end
2015-10-02 07:46:38 -04:00
def ci_yaml_file
return @ci_yaml_file if defined?(@ci_yaml_file)
2017-02-01 18:18:28 -05:00
@ci_yaml_file = project.repository.gitlab_ci_yml_for(sha) rescue nil
2015-10-02 07:46:38 -04:00
end
def has_yaml_errors?
yaml_errors.present?
end
2016-04-29 09:14:38 -04:00
def environments
builds.where.not(environment: nil).success.pluck(:environment).uniq
end
2016-06-23 06:29:11 -04:00
# Manually set the notes for a Ci::Pipeline
# There is no ActiveRecord relation between Ci::Pipeline and notes
# as they are related to a commit sha. This method helps importing
# them using the +Gitlab::ImportExport::RelationFactory+ class.
def notes=(notes)
notes.each do |note|
note[:id] = nil
note[:commit_id] = sha
note[:noteable_id] = self['id']
note.save!
end
end
def notes
Note.for_commit_id(sha)
end
def process!
Ci::ProcessPipelineService.new(project, user).execute(self)
end
def update_status
2016-10-21 09:12:11 -04:00
Gitlab::OptimisticLocking.retry_lock(self) do
2016-09-19 11:20:30 -04:00
case latest_builds_status
when 'pending' then enqueue
when 'running' then run
when 'success' then succeed
when 'failed' then drop
when 'canceled' then cancel
when 'skipped' then skip
when 'manual' then block
end
end
refresh_build_status_cache
end
2016-07-20 07:17:21 -04:00
def predefined_variables
[
{ key: 'CI_PIPELINE_ID', value: id.to_s, public: true }
]
end
def queued_duration
return unless started_at
seconds = (started_at - created_at).to_i
seconds unless seconds.zero?
end
def update_duration
return unless started_at
self.duration = Gitlab::Ci::PipelineDuration.from_pipeline(self)
end
def execute_hooks
2016-08-16 04:00:13 -04:00
data = pipeline_data
project.execute_hooks(data, :pipeline_hooks)
project.execute_services(data, :pipeline_hooks)
end
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
# Merge requests for which the current pipeline is running against
# the merge request's latest commit.
def merge_requests
@merge_requests ||= project.merge_requests
.where(source_branch: self.ref)
.select { |merge_request| merge_request.head_pipeline.try(:id) == self.id }
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
end
def detailed_status(current_user)
Gitlab::Ci::Status::Pipeline::Factory
.new(self, current_user)
.fabricate!
end
def refresh_build_status_cache
Ci::PipelineStatus.new(project, sha: sha, status: status).store_in_cache_if_needed
end
private
def pipeline_data
Gitlab::DataBuilder::Pipeline.build(self)
2016-04-11 10:55:40 -04:00
end
def latest_builds_status
return 'failed' unless yaml_errors.blank?
statuses.latest.status || 'skipped'
2016-04-11 10:55:40 -04:00
end
def keep_around_commits
return unless project
project.repository.keep_around(self.sha)
project.repository.keep_around(self.before_sha)
end
2015-08-25 21:42:46 -04:00
end
end