2020-07-17 00:09:37 +00:00
---
stage: Create
group: Gitaly
2020-11-26 06:09:20 +00:00
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-07-17 00:09:37 +00:00
type: reference
---
2017-09-27 14:59:51 +00:00
2020-07-17 00:09:37 +00:00
# Gitaly developers guide
[Gitaly ](https://gitlab.com/gitlab-org/gitaly ) is a high-level Git RPC service used by GitLab Rails,
Workhorse and GitLab Shell.
2017-09-27 14:59:51 +00:00
2019-06-18 11:11:58 +00:00
## Deep Dive
2021-01-27 03:08:58 +00:00
In May 2019, <!-- vale gitlab.Spelling = NO --> Bob Van Landuyt <!-- vale gitlab.Spelling = YES -->
hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1` )
on the [Gitaly project ](https://gitlab.com/gitlab-org/gitaly ). It included how to contribute to it as a
Ruby developer, and shared domain-specific knowledge with anyone who may work in this part of the
2020-12-11 06:10:17 +00:00
codebase in the future.
2019-06-18 11:11:58 +00:00
2021-01-26 21:09:04 +00:00
You can find the < i class = "fa fa-youtube-play youtube" aria-hidden = "true" ></ i > [recording on YouTube ](https://www.youtube.com/watch?v=BmlEWFS8ORo ), and the slides
2020-04-01 06:07:50 +00:00
on [Google Slides ](https://docs.google.com/presentation/d/1VgRbiYih9ODhcPnL8dS0W98EwFYpJ7GXMPpX-1TM6YE/edit )
and in [PDF ](https://gitlab.com/gitlab-org/create-stage/uploads/a4fdb1026278bda5c1c5bb574379cf80/Create_Deep_Dive__Gitaly_for_Create_Ruby_Devs.pdf ).
Everything covered in this deep dive was accurate as of GitLab 11.11, and while specific details may
2021-01-27 03:08:58 +00:00
have changed, it should still serve as a good introduction.
2019-06-18 11:11:58 +00:00
2019-02-28 03:34:22 +00:00
## Beginner's guide
2019-10-11 06:06:27 +00:00
Start by reading the Gitaly repository's
2021-01-11 00:10:36 +00:00
[Beginner's guide to Gitaly contributions ](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/beginners_guide.md ).
It describes how to set up Gitaly, the various components of Gitaly and what
they do, and how to run its test suites.
2019-02-28 03:34:22 +00:00
2018-04-10 09:31:31 +00:00
## Developing new Git features
2019-03-04 12:15:18 +00:00
To read or write Git data, a request has to be made to Gitaly. This means that
if you're developing a new feature where you need data that's not yet available
in `lib/gitlab/git` changes have to be made to Gitaly.
2018-04-10 09:31:31 +00:00
2021-01-11 00:10:36 +00:00
There should be no new code that touches Git repositories via disk access (for example,
Rugged, `git` , `rm -rf` ) anywhere in the `gitlab` repository. Anything that
needs direct access to the Git repository *must* be implemented in Gitaly, and
exposed via an RPC.
2018-04-10 09:31:31 +00:00
2021-01-11 00:10:36 +00:00
It's often easier to develop a new feature in Gitaly if you make the changes to
2021-01-27 03:08:58 +00:00
GitLab that intends to use the new feature in a separate merge request, to be merged
2021-01-11 00:10:36 +00:00
immediately after the Gitaly one. This allows you to test your changes before
they are merged.
2018-04-10 09:31:31 +00:00
2021-01-11 00:10:36 +00:00
- See [below ](#running-tests-with-a-locally-modified-version-of-gitaly ) for instructions on running GitLab tests with a modified version of Gitaly.
2018-04-10 09:31:31 +00:00
- In GDK run `gdk install` and restart `gdk run` (or `gdk run app` ) to use a locally modified Gitaly version for development
2020-04-08 06:09:54 +00:00
### `gitaly-ruby`
2018-04-10 09:31:31 +00:00
It is possible to implement and test RPC's in Gitaly using Ruby code,
in
2019-10-11 06:06:27 +00:00
[`gitaly-ruby` ](https://gitlab.com/gitlab-org/gitaly/tree/master/ruby ).
2018-04-10 09:31:31 +00:00
This should make it easier to contribute for developers who are less
comfortable writing Go code.
There is documentation for this approach in [the Gitaly
2020-06-03 06:08:34 +00:00
repository](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/ruby_endpoint.md).
2018-04-10 09:31:31 +00:00
2017-09-27 14:59:51 +00:00
## Gitaly-Related Test Failures
If your test-suite is failing with Gitaly issues, as a first step, try running:
```shell
rm -rf tmp/tests/gitaly
```
2020-11-18 15:09:08 +00:00
During RSpec tests, the Gitaly instance writes logs to `gitlab/log/gitaly-test.log` .
2019-05-02 22:47:29 +00:00
2019-02-21 21:07:26 +00:00
## Legacy Rugged code
While Gitaly can handle all Git access, many of GitLab customers still
run Gitaly atop NFS. The legacy Rugged implementation for Git calls may
be faster than the Gitaly RPC due to N+1 Gitaly calls and other
reasons. See [the
2020-05-21 06:08:25 +00:00
issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/57317) for more
2019-02-21 21:07:26 +00:00
details.
Until GitLab has eliminated most of these inefficiencies or the use of
NFS is discontinued for Git data, Rugged implementations of some of the
most commonly-used RPCs can be enabled via feature flags:
2019-07-01 03:36:23 +00:00
- `rugged_find_commit`
- `rugged_get_tree_entries`
- `rugged_tree_entry`
- `rugged_commit_is_ancestor`
- `rugged_commit_tree_entry`
- `rugged_list_commits_by_oid`
2019-02-21 21:07:26 +00:00
A convenience Rake task can be used to enable or disable these flags
all together. To enable:
2020-01-30 15:09:15 +00:00
```shell
2019-02-21 21:07:26 +00:00
bundle exec rake gitlab:features:enable_rugged
```
To disable:
2020-01-30 15:09:15 +00:00
```shell
2019-02-21 21:07:26 +00:00
bundle exec rake gitlab:features:disable_rugged
```
Most of this code exists in the `lib/gitlab/git/rugged_impl` directory.
2020-12-04 21:09:29 +00:00
NOTE:
2021-01-27 03:08:58 +00:00
You should *not* need to add or modify code related to
Rugged unless explicitly discussed with the
[Gitaly Team ](https://gitlab.com/groups/gl-gitaly/group_members ). This code does
2019-02-21 21:07:26 +00:00
NOT work on GitLab.com or other GitLab instances that do not use NFS.
2017-09-30 16:26:23 +00:00
## `TooManyInvocationsError` errors
2017-10-12 08:39:11 +00:00
During development and testing, you may experience `Gitlab::GitalyClient::TooManyInvocationsError` failures.
2020-11-18 15:09:08 +00:00
The `GitalyClient` attempts to block against potential n+1 issues by raising this error
2017-09-30 16:26:23 +00:00
when Gitaly is called more than 30 times in a single Rails request or Sidekiq execution.
2020-11-18 15:09:08 +00:00
As a temporary measure, export `GITALY_DISABLE_REQUEST_LIMITS=1` to suppress the error. This disables the n+1 detection
2017-09-30 16:26:23 +00:00
in your development environment.
Please raise an issue in the GitLab CE or EE repositories to report the issue. Include the labels ~Gitaly
~performance ~"technical debt". Please ensure that the issue contains the full stack trace and error message of the
`TooManyInvocationsError` . Also include any known failing tests if possible.
2020-11-18 15:09:08 +00:00
Isolate the source of the n+1 problem. This is normally a loop that results in Gitaly being called for each
2017-10-12 08:39:11 +00:00
element in an array. If you are unable to isolate the problem, please contact a member
2017-09-30 16:26:23 +00:00
of the [Gitaly Team ](https://gitlab.com/groups/gl-gitaly/group_members ) for assistance.
2021-01-27 03:08:58 +00:00
After the source has been found, wrap it in an `allow_n_plus_1_calls` block, as follows:
2017-09-30 16:26:23 +00:00
```ruby
# n+1: link to n+1 issue
Gitlab::GitalyClient.allow_n_plus_1_calls do
# original code
commits.each { |commit| ... }
end
```
2021-01-27 03:08:58 +00:00
After the code is wrapped in this block, this code path is excluded from n+1 detection.
2017-09-30 16:26:23 +00:00
2017-10-12 08:39:11 +00:00
## Request counts
2019-08-27 08:44:07 +00:00
Commits and other Git data, is now fetched through Gitaly. These fetches can,
2017-10-12 08:39:11 +00:00
much like with a database, be batched. This improves performance for the client
and for Gitaly itself and therefore for the users too. To keep performance stable
and guard performance regressions, Gitaly calls can be counted and the call count
2017-10-17 08:46:05 +00:00
can be tested against. This requires the `:request_store` flag to be set.
2017-10-12 08:39:11 +00:00
```ruby
describe 'Gitaly Request count tests' do
2017-10-17 09:03:11 +00:00
context 'when the request store is activated', :request_store do
it 'correctly counts the gitaly requests made' do
expect { subject }.to change { Gitlab::GitalyClient.get_request_count }.by(10)
end
2017-10-12 08:39:11 +00:00
end
end
```
2018-01-05 11:31:12 +00:00
## Running tests with a locally modified version of Gitaly
2019-08-27 08:44:07 +00:00
Normally, GitLab CE/EE tests use a local clone of Gitaly in
2018-04-10 09:31:31 +00:00
`tmp/tests/gitaly` pinned at the version specified in
2019-10-28 21:06:24 +00:00
`GITALY_SERVER_VERSION` . The `GITALY_SERVER_VERSION` file supports also
2021-01-27 03:08:58 +00:00
branches and SHA to use a custom commit in [the repository ](https://gitlab.com/gitlab-org/gitaly ).
2020-01-23 12:08:38 +00:00
2020-12-04 21:09:29 +00:00
NOTE:
2020-01-23 12:08:38 +00:00
With the introduction of auto-deploy for Gitaly, the format of
`GITALY_SERVER_VERSION` was aligned with Omnibus syntax.
2020-11-18 15:09:08 +00:00
It no longer supports `=revision` , it evaluates the file content as a Git
2021-01-27 03:08:58 +00:00
reference (branch or SHA). Only if it matches a semantic version does it prepend a `v` .
2020-01-23 12:08:38 +00:00
If you want to run tests locally against a modified version of Gitaly you
2018-04-10 09:31:31 +00:00
can replace `tmp/tests/gitaly` with a symlink. This is much faster
2020-11-18 15:09:08 +00:00
because it avoids a Gitaly re-install each time you run `rspec` .
2018-01-05 11:31:12 +00:00
2020-11-26 03:09:17 +00:00
Make sure this directory contains the files `config.toml` and `praefect.config.toml` .
You can copy them from `config.toml.example` and `config.praefect.toml.example` respectively.
After copying, make sure to edit them so everything points to the correct paths.
2018-01-05 11:31:12 +00:00
```shell
rm -rf tmp/tests/gitaly
ln -s /path/to/gitaly tmp/tests/gitaly
```
Make sure you run `make` in your local Gitaly directory before running
2020-11-18 15:09:08 +00:00
tests. Otherwise, Gitaly fails to boot.
2018-01-05 11:31:12 +00:00
If you make changes to your local Gitaly in between test runs you need
to manually run `make` again.
2020-11-18 15:09:08 +00:00
Note that CI tests do not use your locally modified version of
2018-01-05 11:31:12 +00:00
Gitaly. To use a custom Gitaly version in CI you need to update
2021-01-11 00:10:36 +00:00
GITALY_SERVER_VERSION as described at the beginning of this section.
2018-01-05 11:31:12 +00:00
2021-01-27 03:08:58 +00:00
To use a different Gitaly repository, such as if your changes are present
2019-01-23 18:26:57 +00:00
on a fork, you can specify a `GITALY_REPO_URL` environment variable when
running tests:
```shell
GITALY_REPO_URL=https://gitlab.com/nick.thomas/gitaly bundle exec rspec spec/lib/gitlab/git/repository_spec.rb
```
If your fork of Gitaly is private, you can generate a [Deploy Token ](../user/project/deploy_tokens/index.md )
and specify it in the URL:
```shell
GITALY_REPO_URL=https://gitlab+deploy-token-1000:token-here@gitlab.com/nick.thomas/gitaly bundle exec rspec spec/lib/gitlab/git/repository_spec.rb
```
To use a custom Gitaly repository in CI, for instance if you want your
GitLab fork to always use your own Gitaly fork, set `GITALY_REPO_URL`
2019-04-08 12:32:38 +00:00
as a [CI environment variable ](../ci/variables/README.md#gitlab-cicd-environment-variables ).
2019-01-23 18:26:57 +00:00
2020-01-15 09:08:50 +00:00
### Use a locally modified version of Gitaly RPC client
If you are making changes to the RPC client, such as adding a new endpoint or adding a new
parameter to an existing endpoint, follow the guide for
2021-01-27 03:08:58 +00:00
[Gitaly protobuf specifications ](https://gitlab.com/gitlab-org/gitaly/blob/master/proto/README.md ). After pushing
2020-01-15 09:08:50 +00:00
the branch with the changes (`new-feature-branch`, for example):
1. Change the `gitaly` line in the Rails' `Gemfile` to:
```ruby
gem 'gitaly', git: 'https://gitlab.com/gitlab-org/gitaly.git', branch: 'new-feature-branch'
```
1. Run `bundle install` to use the modified RPC client.
2021-01-11 00:10:36 +00:00
Re-run `bundle install` in the `gitlab` project each time the Gitaly branch
changes to embed a new SHA in the `Gemfile.lock` file.
2017-09-27 14:59:51 +00:00
---
[Return to Development documentation ](README.md )
2019-02-28 01:03:04 +00:00
## Wrapping RPCs in Feature Flags
Here are the steps to gate a new feature in Gitaly behind a feature flag.
### Gitaly
1. Create a package scoped flag name:
2020-02-18 18:09:07 +00:00
```golang
2019-02-28 01:03:04 +00:00
var findAllTagsFeatureFlag = "go-find-all-tags"
```
1. Create a switch in the code using the `featureflag` package:
2020-02-18 18:09:07 +00:00
```golang
2019-02-28 01:03:04 +00:00
if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) {
// go implementation
} else {
// ruby implementation
}
```
2019-10-11 06:06:27 +00:00
1. Create Prometheus metrics:
2019-02-28 01:03:04 +00:00
2020-02-18 18:09:07 +00:00
```golang
2019-07-11 22:53:54 +00:00
var findAllTagsRequests = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "gitaly_find_all_tags_requests_total",
Help: "Counter of go vs ruby implementation of FindAllTags",
},
[]string{"implementation"},
2019-02-28 01:03:04 +00:00
)
func init() {
2019-07-11 22:53:54 +00:00
prometheus.Register(findAllTagsRequests)
2019-02-28 01:03:04 +00:00
}
if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) {
2019-07-11 22:53:54 +00:00
findAllTagsRequests.WithLabelValues("go").Inc()
2019-02-28 01:03:04 +00:00
// go implementation
} else {
2019-07-11 22:53:54 +00:00
findAllTagsRequests.WithLabelValues("ruby").Inc()
2019-06-12 15:51:29 +00:00
// ruby implementation
2019-02-28 01:03:04 +00:00
}
```
1. Set headers in tests:
2020-02-18 18:09:07 +00:00
```golang
2019-02-28 01:03:04 +00:00
import (
"google.golang.org/grpc/metadata"
"gitlab.com/gitlab-org/gitaly/internal/featureflag"
)
//...
md := metadata.New(map[string]string{featureflag.HeaderKey(findAllTagsFeatureFlag): "true"})
ctx = metadata.NewOutgoingContext(context.Background(), md)
c, err = client.FindAllTags(ctx, rpcRequest)
require.NoError(t, err)
```
2019-08-27 08:44:07 +00:00
### GitLab Rails
2019-02-28 01:03:04 +00:00
2019-12-06 12:06:21 +00:00
1. Test in a Rails console by setting the feature flag:
2020-12-04 21:09:29 +00:00
NOTE:
2019-12-06 12:06:21 +00:00
Pay attention to the name of the flag and the one used in the Rails console.
There is a difference between them (dashes replaced by underscores and name
2020-02-12 12:09:01 +00:00
prefix is changed). Make sure to prefix all flags with `gitaly_` .
2019-02-28 01:03:04 +00:00
```ruby
2019-12-06 12:06:21 +00:00
Feature.enable('gitaly_go_find_all_tags')
2019-02-28 01:03:04 +00:00
```
2019-12-06 12:06:21 +00:00
### Testing with GDK
To be sure that the flag is set correctly and it goes into Gitaly, you can check
the integration by using GDK:
1. The state of the flag must be observable. To check it, you need to enable it
by fetching the Prometheus metrics:
1. Navigate to GDK's root directory.
1. Make sure you have the proper branch checked out for Gitaly.
1. Recompile it with `make gitaly-setup` and restart the service with `gdk restart gitaly` .
2020-06-03 06:08:34 +00:00
1. Make sure your setup is running: `gdk status | grep praefect` .
1. Check what configuration file is used: `cat ./services/praefect/run | grep praefect` value of the `-config` flag
2019-12-06 12:06:21 +00:00
1. Uncomment `prometheus_listen_addr` in the configuration file and run `gdk restart gitaly` .
1. Make sure that the flag is not enabled yet:
2021-01-27 03:08:58 +00:00
1. Perform whatever action is required to trigger your changes, such as project creation,
submitting commit, or observing history.
2019-12-06 12:06:21 +00:00
1. Check that the list of current metrics has the new counter for the feature flag:
2020-01-30 15:09:15 +00:00
```shell
2020-12-09 06:09:41 +00:00
curl --silent "http://localhost:9236/metrics" | grep go_find_all_tags
2019-12-06 12:06:21 +00:00
```
2021-01-27 03:08:58 +00:00
1. After you observe the metrics for the new feature flag and it increments, you
2019-12-06 12:06:21 +00:00
can enable the new feature:
1. Navigate to GDK's root directory.
1. Start a Rails console:
2020-01-30 15:09:15 +00:00
```shell
2019-12-06 12:06:21 +00:00
bundle install & & bundle exec rails console
```
1. Check the list of feature flags:
```ruby
Feature::Gitaly.server_feature_flags
```
It should be disabled `"gitaly-feature-go-find-all-tags"=>"false"` .
1. Enable it:
```ruby
Feature.enable('gitaly_go_find_all_tags')
```
1. Exit the Rails console and perform whatever action is required to trigger
2021-01-27 03:08:58 +00:00
your changes, such as project creation, submitting commit, or observing history.
2019-12-06 12:06:21 +00:00
1. Verify the feature is on by observing the metrics for it:
2020-01-30 15:09:15 +00:00
```shell
2020-12-09 06:09:41 +00:00
curl --silent "http://localhost:9236/metrics" | grep go_find_all_tags
2019-12-06 12:06:21 +00:00
```