gitlab-org--gitlab-foss/app/models/ci/pipeline.rb

365 lines
8.9 KiB
Ruby
Raw Normal View History

2015-08-25 21:42:46 -04:00
module Ci
class Pipeline < ActiveRecord::Base
2015-08-25 21:42:46 -04:00
extend Ci::Model
2016-08-24 22:55:32 -04:00
include HasStatus
include Importable
include AfterCommitQueue
2015-09-24 11:09:33 -04:00
2016-06-03 06:00:16 -04:00
self.table_name = 'ci_commits'
belongs_to :project, foreign_key: :gl_project_id
2016-07-15 09:42:29 -04:00
belongs_to :user
2016-06-03 07:58:35 -04:00
has_many :statuses, class_name: 'CommitStatus', foreign_key: :commit_id
has_many :builds, foreign_key: :commit_id
has_many :trigger_requests, dependent: :destroy, foreign_key: :commit_id
2015-08-25 21:42:46 -04:00
validates_presence_of :sha, unless: :importing?
validates_presence_of :ref, unless: :importing?
validates_presence_of :status, unless: :importing?
validate :valid_commit_sha, unless: :importing?
2015-08-25 21:42:46 -04:00
after_create :keep_around_commits, unless: :importing?
2016-04-11 10:55:40 -04:00
state_machine :status, initial: :created do
2016-08-12 07:57:58 -04:00
event :enqueue do
2016-08-12 06:23:47 -04:00
transition created: :pending
transition [:success, :failed, :canceled, :skipped] => :running
end
event :run do
2016-10-20 03:33:44 -04:00
transition any - [:running] => :running
end
event :skip do
2016-10-20 03:33:44 -04:00
transition any - [:skipped] => :skipped
end
event :drop do
2016-10-20 03:33:44 -04:00
transition any - [:failed] => :failed
end
event :succeed do
2016-10-20 03:33:44 -04:00
transition any - [:success] => :success
end
event :cancel do
2016-10-20 03:33:44 -04:00
transition any - [:canceled] => :canceled
end
2016-10-14 13:08:48 -04:00
# IMPORTANT
# Do not add any operations to this state_machine
# Create a separate worker for each new operation
before_transition [:created, :pending] => :running do |pipeline|
pipeline.started_at = Time.now
end
before_transition any => [:success, :failed, :canceled] do |pipeline|
pipeline.finished_at = Time.now
pipeline.update_duration
end
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
after_transition [:created, :pending] => :running do |pipeline|
2016-10-14 13:08:48 -04:00
pipeline.run_after_commit { PipelineMetricsWorker.perform_async(id) }
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
end
after_transition any => [:success] do |pipeline|
2016-10-14 13:08:48 -04:00
pipeline.run_after_commit { PipelineMetricsWorker.perform_async(id) }
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
end
after_transition [:created, :pending, :running] => :success do |pipeline|
pipeline.run_after_commit { PipelineSuccessWorker.perform_async(id) }
end
after_transition do |pipeline, transition|
2016-10-12 07:44:33 -04:00
next if transition.loopback?
pipeline.run_after_commit do
PipelineHooksWorker.perform_async(id)
end
end
after_transition any => [:success, :failed] do |pipeline|
pipeline.run_after_commit do
PipelineNotificationWorker.perform_async(pipeline.id)
end
end
end
# ref can't be HEAD or SHA, can only be branch/tag name
def self.latest_successful_for(ref)
where(ref: ref).order(id: :desc).success.first
end
2015-08-25 21:42:46 -04:00
def self.truncate_sha(sha)
sha[0...8]
end
def self.total_duration
2016-08-15 06:50:02 -04:00
where.not(duration: nil).sum(:duration)
end
def stages_count
statuses.select(:stage).distinct.count
end
2016-12-06 08:49:37 -05:00
def stages_name
2016-12-07 06:47:06 -05:00
statuses.order(:stage_idx).distinct.
pluck(:stage, :stage_idx).map(&:first)
2016-12-06 08:49:37 -05:00
end
def stages
status_sql = statuses.latest.where('stage=sg.stage').status_sql
stages_query = statuses.group('stage').select(:stage)
.order('max(stage_idx)')
stages_with_statuses = CommitStatus.from(stages_query, :sg).
pluck('sg.stage', status_sql)
stages_with_statuses.map do |stage|
Ci::Stage.new(self, name: stage.first, status: stage.last)
end
end
def artifacts
builds.latest.with_artifacts_not_expired
end
2015-09-28 07:35:26 -04:00
def project_id
project.id
2015-09-28 07:14:34 -04:00
end
# For now the only user who participates is the user who triggered
def participants(_current_user = nil)
Array(user)
end
2015-08-25 21:42:46 -04:00
def valid_commit_sha
2015-12-11 11:57:04 -05:00
if self.sha == Gitlab::Git::BLANK_SHA
2015-08-25 21:42:46 -04:00
self.errors.add(:sha, " cant be 00000000 (branch removal)")
end
end
def git_author_name
commit.try(:author_name)
2015-08-25 21:42:46 -04:00
end
def git_author_email
commit.try(:author_email)
2015-08-25 21:42:46 -04:00
end
def git_commit_message
commit.try(:message)
2015-08-25 21:42:46 -04:00
end
def git_commit_title
commit.try(:title)
end
2015-08-25 21:42:46 -04:00
def short_sha
Ci::Pipeline.truncate_sha(sha)
2015-08-25 21:42:46 -04:00
end
def commit
2015-12-04 06:55:23 -05:00
@commit ||= project.commit(sha)
2015-08-25 21:42:46 -04:00
rescue
nil
end
2016-03-31 13:51:28 -04:00
def branch?
!tag?
end
def manual_actions
builds.latest.manual_actions
2016-07-16 12:39:58 -04:00
end
2016-03-31 13:51:28 -04:00
def retryable?
builds.latest.failed_or_canceled.any?(&:retryable?)
2016-03-31 13:51:28 -04:00
end
def cancelable?
statuses.cancelable.any?
end
2016-05-09 19:26:13 -04:00
def cancel_running
Gitlab::OptimisticLocking.retry_lock(
statuses.cancelable) do |cancelable|
cancelable.each(&:cancel)
end
2016-05-09 19:26:13 -04:00
end
def retry_failed(user)
Gitlab::OptimisticLocking.retry_lock(
builds.latest.failed_or_canceled) do |failed_or_canceled|
failed_or_canceled.select(&:retryable?).each do |build|
Ci::Build.retry(build, user)
end
end
2016-05-09 19:26:13 -04:00
end
def mark_as_processable_after_stage(stage_idx)
2016-08-19 09:02:35 -04:00
builds.skipped.where('stage_idx > ?', stage_idx).find_each(&:process)
end
2016-04-13 07:01:08 -04:00
def latest?
return false unless ref
commit = project.commit(ref)
return false unless commit
commit.sha == sha
end
2016-04-12 13:57:22 -04:00
def triggered?
trigger_requests.any?
end
2015-10-06 06:01:16 -04:00
def retried
@retried ||= (statuses.order(id: :desc) - statuses.latest)
2015-08-25 21:42:46 -04:00
end
def coverage
2016-04-13 14:51:03 -04:00
coverage_array = statuses.latest.map(&:coverage).compact
2015-12-04 06:55:23 -05:00
if coverage_array.size >= 1
'%.2f' % (coverage_array.reduce(:+) / coverage_array.size)
2015-08-25 21:42:46 -04:00
end
end
def config_builds_attributes
return [] unless config_processor
config_processor.
builds_for_ref(ref, tag?, trigger_requests.first).
sort_by { |build| build[:stage_idx] }
end
2016-07-14 10:58:05 -04:00
def has_warnings?
builds.latest.failed_but_allowed.any?
end
2015-08-25 21:42:46 -04:00
def config_processor
return nil unless ci_yaml_file
return @config_processor if defined?(@config_processor)
@config_processor ||= begin
Ci::GitlabCiYamlProcessor.new(ci_yaml_file, project.path_with_namespace)
rescue Ci::GitlabCiYamlProcessor::ValidationError, Psych::SyntaxError => e
self.yaml_errors = e.message
nil
rescue
self.yaml_errors = 'Undefined error'
nil
end
2015-08-25 21:42:46 -04:00
end
2015-10-02 07:46:38 -04:00
def ci_yaml_file
return @ci_yaml_file if defined?(@ci_yaml_file)
2016-02-02 08:35:09 -05:00
@ci_yaml_file ||= begin
blob = project.repository.blob_at(sha, '.gitlab-ci.yml')
blob.load_all_data!(project.repository)
blob.data
rescue
nil
2016-02-02 08:35:09 -05:00
end
2015-10-02 07:46:38 -04:00
end
2016-04-29 09:14:38 -04:00
def environments
builds.where.not(environment: nil).success.pluck(:environment).uniq
end
2016-06-23 06:29:11 -04:00
# Manually set the notes for a Ci::Pipeline
# There is no ActiveRecord relation between Ci::Pipeline and notes
# as they are related to a commit sha. This method helps importing
# them using the +Gitlab::ImportExport::RelationFactory+ class.
def notes=(notes)
notes.each do |note|
note[:id] = nil
note[:commit_id] = sha
note[:noteable_id] = self['id']
note.save!
end
end
def notes
Note.for_commit_id(sha)
end
def process!
Ci::ProcessPipelineService.new(project, user).execute(self)
end
def update_status
2016-10-21 09:12:11 -04:00
Gitlab::OptimisticLocking.retry_lock(self) do
2016-09-19 11:20:30 -04:00
case latest_builds_status
when 'pending' then enqueue
when 'running' then run
when 'success' then succeed
when 'failed' then drop
when 'canceled' then cancel
when 'skipped' then skip
end
end
end
2016-07-20 07:17:21 -04:00
def predefined_variables
[
{ key: 'CI_PIPELINE_ID', value: id.to_s, public: true }
]
end
def queued_duration
return unless started_at
seconds = (started_at - created_at).to_i
seconds unless seconds.zero?
end
def update_duration
return unless started_at
self.duration = Gitlab::Ci::PipelineDuration.from_pipeline(self)
end
def execute_hooks
2016-08-16 04:00:13 -04:00
data = pipeline_data
project.execute_hooks(data, :pipeline_hooks)
project.execute_services(data, :pipeline_hooks)
end
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
# Merge requests for which the current pipeline is running against
# the merge request's latest commit.
def merge_requests
@merge_requests ||= project.merge_requests
.where(source_branch: self.ref)
.select { |merge_request| merge_request.head_pipeline.try(:id) == self.id }
Improve performance of the cycle analytics page. 1. These changes bring down page load time for 100 issues from more than a minute to about 1.5 seconds. 2. This entire commit is composed of these types of performance enhancements: - Cache relevant data in `IssueMetrics` wherever possible. - Cache relevant data in `MergeRequestMetrics` wherever possible. - Preload metrics 3. Given these improvements, we now only need to make 4 SQL calls: - Load all issues - Load all merge requests - Load all metrics for the issues - Load all metrics for the merge requests 4. A list of all the data points that are now being pre-calculated: a. The first time an issue is mentioned in a commit - In `GitPushService`, find all issues mentioned by the given commit using `ReferenceExtractor`. Set the `first_mentioned_in_commit_at` flag for each of them. - There seems to be a (pre-existing) bug here - files (and therefore commits) created using the Web CI don't have cross-references created, and issues are not closed even when the commit title is "Fixes #xx". b. The first time a merge request is deployed to production When a `Deployment` is created, find all merge requests that were merged in before the deployment, and set the `first_deployed_to_production_at` flag for each of them. c. The start / end time for a merge request pipeline Hook into the `Pipeline` state machine. When the `status` moves to `running`, find the merge requests whose tip commit matches the pipeline, and record the `latest_build_started_at` time for each of them. When the `status` moves to `success`, record the `latest_build_finished_at` time. d. The merge requests that close an issue - This was a big cause of the performance problems we were having with Cycle Analytics. We need to use `ReferenceExtractor` to make this calculation, which is slow when we have to run it on a large number of merge requests. - When a merge request is created, updated, or refreshed, find the issues it closes, and create an instance of `MergeRequestsClosingIssues`, which acts as a join model between merge requests and issues. - If a `MergeRequestsClosingIssues` instance links a merge request and an issue, that issue closes that merge request. 5. The `Queries` module was changed into a class, so we can cache the results of `issues` and `merge_requests_closing_issues` across various cycle analytics stages. 6. The code added in this commit is untested. Tests will be added in the next commit.
2016-09-15 04:59:36 -04:00
end
def detailed_status(current_user)
Gitlab::Ci::Status::Pipeline::Factory
.new(self, current_user)
.fabricate!
end
private
def pipeline_data
Gitlab::DataBuilder::Pipeline.build(self)
2016-04-11 10:55:40 -04:00
end
def latest_builds_status
return 'failed' unless yaml_errors.blank?
statuses.latest.status || 'skipped'
2016-04-11 10:55:40 -04:00
end
def keep_around_commits
return unless project
project.repository.keep_around(self.sha)
project.repository.keep_around(self.before_sha)
end
2015-08-25 21:42:46 -04:00
end
end