2018-03-27 16:47:49 +00:00
# Job traces (logs)
2018-06-22 15:49:08 +00:00
Job traces are sent by GitLab Runner while it's processing a job. You can see
traces in job pages, pipelines, email notifications, etc.
2018-03-27 16:47:49 +00:00
2018-06-22 15:49:08 +00:00
## Data flow
In general, there are two states in job traces: "live trace" and "archived trace".
2018-06-30 03:26:19 +00:00
In the following table you can see the phases a trace goes through.
2018-06-22 15:49:08 +00:00
| Phase | State | Condition | Data flow | Stored path |
| ----- | ----- | --------- | --------- | ----------- |
2019-03-25 10:11:55 +00:00
| 1: patching | Live trace | When a job is running | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
| 2: overwriting | Live trace | When a job is finished | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
| 3: archiving | Archived trace | After a job is finished | Sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`|
2018-09-04 13:41:07 +00:00
| 4: uploading | Archived trace | After a trace is archived | Sidekiq moves archived trace to [object storage ](#uploading-traces-to-object-storage ) (if configured) |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`|
2018-06-22 15:49:08 +00:00
2018-06-30 03:26:19 +00:00
The `ROOT_PATH` varies per your environment. For Omnibus GitLab it
2019-03-25 10:11:55 +00:00
would be `/var/opt/gitlab` , whereas for installations from source
2018-06-22 15:49:08 +00:00
it would be `/home/git/gitlab` .
## Changing the job traces local location
2018-03-27 16:47:49 +00:00
To change the location where the job logs will be stored, follow the steps below.
**In Omnibus installations:**
2019-07-09 07:16:17 +00:00
1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line:
2018-03-27 16:47:49 +00:00
2019-07-09 07:16:17 +00:00
```ruby
gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds'
```
2018-03-27 16:47:49 +00:00
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
---
**In installations from source:**
1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
2019-07-09 07:16:17 +00:00
```yaml
gitlab_ci:
# The location where build traces are stored (default: builds/).
# Relative paths are relative to Rails.root.
builds_path: path/to/builds/
```
2018-03-27 16:47:49 +00:00
1. Save the file and [restart GitLab][] for the changes to take effect.
[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab"
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
## Uploading traces to object storage
2018-06-12 11:49:32 +00:00
2018-08-16 14:28:47 +00:00
Archived traces are considered as [job artifacts ](job_artifacts.md ).
Therefore, when you [set up the object storage integration ](job_artifacts.md#object-storage-settings ),
2018-06-22 15:49:08 +00:00
job traces are automatically migrated to it along with the other job artifacts.
2018-06-12 11:49:32 +00:00
2018-08-16 14:28:47 +00:00
See "Phase 4: uploading" in [Data flow ](#data-flow ) to learn about the process.
## How to archive legacy job trace files
Legacy job traces, which were created before GitLab 10.5, were not archived regularly.
It's the same state with the "2: overwriting" in the above [Data flow ](#data-flow ).
To archive those legacy job traces, please follow the instruction below.
1. Execute the following command
2019-07-09 07:16:17 +00:00
```bash
gitlab-rake gitlab:traces:archive
```
2018-08-16 14:28:47 +00:00
2019-07-09 07:16:17 +00:00
After you executed this task, GitLab instance queues up Sidekiq jobs (asynchronous processes)
for migrating job trace files from local storage to object storage.
It could take time to complete the all migration jobs. You can check the progress by the following command
2018-08-16 14:28:47 +00:00
2019-07-09 07:16:17 +00:00
```bash
sudo gitlab-rails console
```
2018-08-16 14:28:47 +00:00
2019-07-09 07:16:17 +00:00
```bash
[1] pry(main)> Sidekiq::Stats.new.queues['pipeline_background:archive_trace']
=> 100
```
2018-08-16 14:28:47 +00:00
2019-07-09 07:16:17 +00:00
If the count becomes zero, the archiving processes are done
2018-08-16 14:28:47 +00:00
## How to migrate archived job traces to object storage
2018-09-04 13:41:07 +00:00
> [Introduced][ce-21193] in GitLab 11.3.
2018-08-16 14:28:47 +00:00
If job traces have already been archived into local storage, and you want to migrate those traces to object storage, please follow the instruction below.
1. Ensure [Object storage integration for Job Artifacts ](job_artifacts.md#object-storage-settings ) is enabled
1. Execute the following command
2019-07-10 18:23:55 +00:00
```bash
gitlab-rake gitlab:traces:migrate
```
2018-08-16 14:28:47 +00:00
## How to remove job traces
There isn't a way to automatically expire old job logs, but it's safe to remove
them if they're taking up too much space. If you remove the logs manually, the
job output in the UI will be empty.
2018-06-12 11:49:32 +00:00
2018-05-07 07:21:09 +00:00
## New live trace architecture
2018-06-22 15:49:08 +00:00
> [Introduced][ce-18169] in GitLab 10.4.
2018-06-12 11:49:32 +00:00
> [Announced as General availability][ce-46097] in GitLab 11.0.
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
NOTE: **Note:**
This feature is off by default. Check below how to [enable/disable ](#enabling-live-trace ) it.
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
By combining the process with object storage settings, we can completely bypass
the local file storage. This is a useful option if GitLab is installed as
cloud-native, for example on Kubernetes.
2018-06-12 11:49:32 +00:00
2018-06-22 15:49:08 +00:00
The data flow is the same as described in the [data flow section ](#data-flow )
with one change: _the stored path of the first two phases is different_ . This new live
2018-07-03 04:31:32 +00:00
trace architecture stores chunks of traces in Redis and a persistent store (object storage or database) instead of
2018-06-22 15:49:08 +00:00
file storage. Redis is used as first-class storage, and it stores up-to 128KB
2018-07-03 04:31:32 +00:00
of data. Once the full chunk is sent, it is flushed a persistent store, either object storage(temporary directory) or database.
After a while, the data in Redis and a persitent store will be archived to [object storage ](#uploading-traces-to-object-storage ).
2018-06-12 11:49:32 +00:00
2018-06-22 15:49:08 +00:00
The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState` .
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
Here is the detailed data flow:
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
1. GitLab Runner picks a job from GitLab
1. GitLab Runner sends a piece of trace to GitLab
1. GitLab appends the data to Redis
2018-07-03 04:31:32 +00:00
1. Once the data in Redis reach 128KB, the data is flushed to a persistent store (object storage or the database).
2018-06-22 15:49:08 +00:00
1. The above steps are repeated until the job is finished.
1. Once the job is finished, GitLab schedules a Sidekiq worker to archive the trace.
1. The Sidekiq worker archives the trace to object storage and cleans up the trace
2018-07-03 04:31:32 +00:00
in Redis and a persistent store (object storage or the database).
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
### Enabling live trace
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
The following commands are to be issues in a Rails console:
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
```sh
# Omnibus GitLab
gitlab-rails console
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
# Installation from source
cd /home/git/gitlab
sudo -u git -H bin/rails console RAILS_ENV=production
```
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
**To check if live trace is enabled:**
2018-05-07 07:21:09 +00:00
```ruby
Feature.enabled?('ci_enable_live_trace')
```
2018-06-22 15:49:08 +00:00
**To enable live trace:**
2018-05-07 07:21:09 +00:00
```ruby
Feature.enable('ci_enable_live_trace')
```
2018-06-22 15:49:08 +00:00
NOTE: **Note:**
The transition period will be handled gracefully. Upcoming traces will be
generated with the new architecture, and on-going live traces will stay with the
legacy architecture, which means that on-going live traces won't be forcibly
re-generated with the new architecture.
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
**To disable live trace:**
2018-05-07 07:21:09 +00:00
```ruby
Feature.disable('ci_enable_live_trace')
```
2018-06-22 15:49:08 +00:00
NOTE: **Note:**
The transition period will be handled gracefully. Upcoming traces will be generated
with the legacy architecture, and on-going live traces will stay with the new
architecture, which means that on-going live traces won't be forcibly re-generated
with the legacy architecture.
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
### Potential implications
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
In some cases, having data stored on Redis could incur data loss:
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
1. **Case 1: When all data in Redis are accidentally flushed**
2018-09-06 16:52:18 +00:00
- On going live traces could be recovered by re-sending traces (this is
supported by all versions of the GitLab Runner).
- Finished jobs which have not archived live traces will lose the last part
(~128KB) of trace data.
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
1. **Case 2: When Sidekiq workers fail to archive (e.g., there was a bug that
prevents archiving process, Sidekiq inconsistency, etc.)**
2018-09-06 16:52:18 +00:00
- Currently all trace data in Redis will be deleted after one week. If the
Sidekiq workers can't finish by the expiry date, the part of trace data will be lost.
2018-05-07 07:21:09 +00:00
2018-06-22 15:49:08 +00:00
Another issue that might arise is that it could consume all memory on the Redis
instance. If the number of jobs is 1000, 128MB (128KB * 1000) is consumed.
2018-05-07 08:40:40 +00:00
2018-06-22 15:49:08 +00:00
Also, it could pressure the database replication lag. `INSERT` s are generated to
indicate that we have trace chunk. `UPDATE` s with 128KB of data is issued once we
receive multiple chunks.
2018-05-07 07:21:09 +00:00
2019-09-18 14:02:45 +00:00
[ce-18169]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/18169
[ce-21193]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/21193
[ce-46097]: https://gitlab.com/gitlab-org/gitlab-foss/issues/46097