info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Log system **(FREE SELF)**
GitLab has an advanced log system where everything is logged, so you
can analyze your instance using various system log files. In addition to
system log files, GitLab Enterprise Edition provides [Audit Events](../audit_events.md).
System log files are typically plain text in a standard log file format.
This guide talks about how to read and use these system log files.
Read more about the log system and using the logs:
- [Customize logging on Omnibus GitLab installations](https://docs.gitlab.com/omnibus/settings/logs.html)
including adjusting log retention, log forwarding,
switching logs from JSON to plain text logging, and more.
- [How to parse and analyze JSON logs](../logs/log_parsing.md).
## Log Levels
Each log message has an assigned log level that indicates its importance and verbosity.
Each logger has an assigned minimum log level.
A logger emits a log message only if its log level is equal to or above the minimum log level.
The following log levels are supported:
| Level | Name |
|:------|:----------|
| 0 | `DEBUG` |
| 1 | `INFO` |
| 2 | `WARN` |
| 3 | `ERROR` |
| 4 | `FATAL` |
| 5 | `UNKNOWN` |
GitLab loggers emit all log messages because they are set to `DEBUG` by default.
### Override default log level
You can override the minimum log level for GitLab loggers using the `GITLAB_LOG_LEVEL` environment variable.
Valid values are either a value of `0` to `5`, or the name of the log level.
Example:
```shell
GITLAB_LOG_LEVEL=info
```
For some services, other log levels are in place that are not affected by this setting.
Some of these services have their own environment variables to override the log level. For example:
CACHE (0.0ms) SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."source_id" = $1 AND "members"."source_type" = $2 AND "members"."user_id" = 1 ORDER BY "members"."created_at" DESC, "members"."id" DESC LIMIT 1 [["source_id", 18], ["source_type", "Project"]]
CACHE (0.0ms) SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members".
(1.4ms) SELECT COUNT(*) FROM "merge_requests" WHERE "merge_requests"."target_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened')) [["target_project_id", 18]]
For GitLab 12.5 and earlier, the file is at `/var/log/gitlab/gitlab-shell/gitlab-shell.log`.
Example log entries:
```plaintext
I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>.
I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory and symlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git.
```
User clone/fetch activity using SSH transport appears in this log as
`executing git command <gitaly-upload-pack...`.
## Gitaly Logs
This file is in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/).
`runit` is packaged with Omnibus GitLab and a brief explanation of its purpose
is available [in the Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/architecture/#runit).
[Log files are rotated](http://smarden.org/runit/svlogd.8.html), renamed in
Unix timestamp format, and `gzip`-compressed (like `@1584057562.s`).
### `grpc.log`
This file is at `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab
packages. Native [gRPC](https://grpc.io/) logging used by Gitaly.
### `gitaly_ruby_json.log`
> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2678) in GitLab 13.6.
This file is at `/var/log/gitlab/gitaly/gitaly_ruby_json.log` and is
produced by [`gitaly-ruby`](../gitaly/reference.md#gitaly-ruby). It contains an
access log of gRPC calls made by Gitaly to `gitaly-ruby`.
### `gitaly_hooks.log`
This file is at `/var/log/gitlab/gitaly/gitaly_hooks.log` and is
produced by `gitaly-hooks` command. It also contains records about
failures received during processing of the responses from GitLab API.
## Puma Logs
### `puma_stdout.log`
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/puma/puma_stdout.log`
- Installations from source: `/home/git/gitlab/log/puma_stdout.log`
### `puma_stderr.log`
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/puma/puma_stderr.log`
- Installations from source: `/home/git/gitlab/log/puma_stderr.log`
## `repocheck.log`
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/repocheck.log`
- Installations from source: `/home/git/gitlab/log/repocheck.log`
It logs information whenever a [repository check is run](../repository_checks.md)
on a project.
## `importer.log`
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/importer.log`
- Installations from source: `/home/git/gitlab/log/importer.log`
It logs the progress of the import process.
## `exporter.log`
> Introduced in GitLab 13.1.
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/exporter.log`
- Installations from source: `/home/git/gitlab/log/exporter.log`
It logs the progress of the export process.
## `features_json.log`
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 13.7.
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/features_json.log`
- Installations from source: `/home/git/gitlab/log/features_json.log`
The modification events from [Feature flags in development of GitLab](../../development/feature_flags/index.md)
- In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and later,
user ID and username, if available.
## `graphql_json.log`
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 12.0.
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/graphql_json.log`
- Installations from source: `/home/git/gitlab/log/graphql_json.log`
GraphQL queries are recorded in the file. For example:
```json
{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration_s":7}
```
## `migrations.log`
> Introduced in GitLab 12.3.
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/migrations.log`
- Installations from source: `/home/git/gitlab/log/migrations.log`
## `mail_room_json.log` (default)
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6.
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/mailroom/current`
- Installations from source: `/home/git/gitlab/log/mail_room_json.log`
This structured log file records internal activity in the `mail_room` gem.
Its name and path are configurable, so the name and path may not match the above.
## Reconfigure logs
Reconfigure log files are in `/var/log/gitlab/reconfigure` for Omnibus GitLab
packages. Installations from source don't have reconfigure logs. A reconfigure log
is populated whenever `gitlab-ctl reconfigure` is run manually or as part of an upgrade.
Reconfigure logs files are named according to the UNIX timestamp of when the reconfigure
was initiated, such as `1509705644.log`
## `sidekiq_exporter.log` and `web_exporter.log`
If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq
starts a Web server and listen to the defined port (default:
`8082`). By default, Sidekiq Exporter access logs are disabled but can
be enabled based on your installation method:
- Omnibus GitLab: Use the `sidekiq['exporter_log_enabled'] = true`
option in `/etc/gitlab/gitlab.rb`
- Installations from source: Use the `sidekiq_exporter.log_enabled` option
in `gitlab.yml`
When enabled, depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log`
- Installations from source: `/home/git/gitlab/log/sidekiq_exporter.log`
If Prometheus metrics and the Web Exporter are both enabled, Puma
starts a Web server and listen to the defined port (default: `8083`), and access logs
are generated in a location based on your installation method:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/web_exporter.log`
- Installations from source: `/home/git/gitlab/log/web_exporter.log`
For Omnibus GitLab installations, Mattermost logs are in these locations:
-`/var/log/gitlab/mattermost/mattermost.log`
-`/var/log/gitlab/mattermost/current`
## Workhorse Logs
For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`.
## PostgreSQL Logs
For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`.
## Prometheus Logs
For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/current`.
## Redis Logs
For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/current`.
## Alertmanager Logs
For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/current`.
<!-- vale gitlab.Spelling = NO -->
## Crond Logs
For Omnibus GitLab installations, crond logs are in `/var/log/gitlab/crond/`.
<!-- vale gitlab.Spelling = YES -->
## Grafana Logs
For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/current`.
## LogRotate Logs
For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/current`.
## GitLab Monitor Logs
For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gitlab-monitor/`.
## GitLab Exporter
For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/current`.
## GitLab agent server
For Omnibus GitLab installations, GitLab agent server logs are
in `/var/log/gitlab/gitlab-kas/current`.
## Praefect Logs
For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect/`.
GitLab also tracks [Prometheus metrics for Praefect](../gitaly/monitoring.md#monitor-gitaly-cluster).
## Backup log
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63832) in GitLab 14.1.
For Omnibus installations, the backup log is located at `/var/log/gitlab/gitlab-rails/backup_json.log`.
This log is populated when a [GitLab backup is created](../../raketasks/backup_restore.md). You can use this log to understand how the backup process performed.
## Performance bar stats
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7.
Depending on your installation method, this file is located at:
- Omnibus GitLab: `/var/log/gitlab/gitlab-rails/performance_bar_json.log`
- Installations from source: `/home/git/gitlab/log/performance_bar_json.log`
Performance bar statistics (currently only duration of SQL queries) are recorded