2020-08-05 11:09:59 -04:00
---
stage: Configure
group: Configure
2020-11-26 01:09:20 -05:00
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
2020-08-05 11:09:59 -04:00
---
2021-02-08 13:09:49 -05:00
# Kubernetes integration - development guidelines **(FREE)**
2019-01-23 15:48:38 -05:00
2020-12-10 16:10:15 -05:00
This document provides various guidelines when developing for the GitLab
2019-01-23 15:48:38 -05:00
[Kubernetes integration ](../user/project/clusters/index.md ).
## Development
### Architecture
Some Kubernetes operations, such as creating restricted project
namespaces are performed on the GitLab Rails application. These
2020-09-21 23:09:49 -04:00
operations are performed using a [client library ](#client-library ),
and carry an element of risk. The operations are
run as the same user running the GitLab Rails application. For more information,
read the [security ](#security ) section below.
2019-01-23 15:48:38 -05:00
Some Kubernetes operations, such as installing cluster applications are
performed on one-off pods on the Kubernetes cluster itself. These
2020-09-21 23:09:49 -04:00
installation pods are named `install-<application_name>` and
2019-01-23 15:48:38 -05:00
are created within the `gitlab-managed-apps` namespace.
In terms of code organization, we generally add objects that represent
Kubernetes resources in
2019-09-18 10:02:45 -04:00
[`lib/gitlab/kubernetes` ](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/lib/gitlab/kubernetes ).
2019-01-23 15:48:38 -05:00
### Client library
We use the [`kubeclient` ](https://rubygems.org/gems/kubeclient ) gem to
perform Kubernetes API calls. As the `kubeclient` gem does not support
2020-09-21 23:09:49 -04:00
different API Groups (such as `apis/rbac.authorization.k8s.io` ) from a
2019-01-23 15:48:38 -05:00
single client, we have created a wrapper class,
2019-09-18 14:06:14 -04:00
[`Gitlab::Kubernetes::KubeClient` ](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb )
2020-09-21 23:09:49 -04:00
that enable you to achieve this.
2019-01-23 15:48:38 -05:00
2020-09-21 23:09:49 -04:00
Selected Kubernetes API groups are supported. Do add support
2019-01-23 15:48:38 -05:00
for new API groups or methods to
2019-09-18 14:06:14 -04:00
[`Gitlab::Kubernetes::KubeClient` ](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb )
2019-01-23 15:48:38 -05:00
if you need to use them. New API groups or API group versions can be
2020-09-21 23:09:49 -04:00
added to `SUPPORTED_API_GROUPS` - internally, this creates an
2019-01-23 15:48:38 -05:00
internal client for that group. New methods can be added as a delegation
to the relevant internal client.
### Performance considerations
2020-09-21 23:09:49 -04:00
All calls to the Kubernetes API must be in a background process. Don't
perform Kubernetes API calls within a web request. This blocks
Unicorn, and can lead to a denial-of-service (DoS) attack in GitLab as
2019-01-23 15:48:38 -05:00
the Kubernetes cluster response times are outside of our control.
The easiest way to ensure your calls happen a background process is to
2019-10-11 02:06:27 -04:00
delegate any such work to happen in a [Sidekiq worker ](sidekiq_style_guide.md ).
2019-01-23 15:48:38 -05:00
2020-09-21 23:09:49 -04:00
You may want to make calls to Kubernetes and return the response, but a background
worker isn't a good fit. Consider using
[reactive caching ](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb ).
2019-01-23 15:48:38 -05:00
For example:
```ruby
def calculate_reactive_cache!
{ pods: cluster.platform_kubernetes.kubeclient.get_pods }
end
def pods
with_reactive_cache do |data|
data[:pods]
end
end
```
### Testing
2020-09-21 23:09:49 -04:00
We have some WebMock stubs in
2019-09-18 14:06:14 -04:00
[`KubernetesHelpers` ](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/support/helpers/kubernetes_helpers.rb )
2019-01-23 15:48:38 -05:00
which can help with mocking out calls to Kubernetes API in your tests.
2019-10-31 23:06:26 -04:00
### Amazon EKS integration
This section outlines the process for allowing a GitLab instance to create EKS clusters.
The following prerequisites are required:
2020-09-21 23:09:49 -04:00
A `Customer` AWS account. The EKS cluster is created in this account. The following
resources must be present:
2019-10-31 23:06:26 -04:00
- A provisioning role that has permissions to create the cluster
and associated resources. It must list the `GitLab` AWS account
as a trusted entity.
- A VPC, management role, security group, and subnets for use by the cluster.
A `GitLab` AWS account. This is the account which performs
the provisioning actions. The following resources must be present:
- A service account with permissions to assume the provisioning
role in the `Customer` account above.
- Credentials for this service account configured in GitLab via
2019-11-06 22:06:16 -05:00
the `kubernetes` section of `gitlab.yml` .
2019-10-31 23:06:26 -04:00
The process for creating a cluster is as follows:
2019-11-06 22:06:16 -05:00
1. Using the `:provision_role_external_id` , GitLab assumes the role provided
by `:provision_role_arn` and stores a set of temporary credentials on the
2019-10-31 23:06:26 -04:00
provider record. By default these credentials are valid for one hour.
1. A CloudFormation stack is created, based on the
[`AWS CloudFormation EKS template` ](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/aws/cloudformation/eks_cluster.yaml ).
This triggers creation of all resources required for an EKS cluster.
1. GitLab polls the status of the stack until all resources are ready,
which takes somewhere between 10 and 15 minutes in most cases.
1. When the stack is ready, GitLab stores the cluster details and generates
another set of temporary credentials, this time to allow connecting to
2020-09-21 23:09:49 -04:00
the cluster via `kubeclient` . These credentials are valid for one minute.
2019-10-31 23:06:26 -04:00
1. GitLab configures the worker nodes so that they are able to authenticate
to the cluster, and creates a service account for itself for future operations.
1. Credentials that are no longer required are removed. This deletes the following
attributes:
2019-11-06 22:06:16 -05:00
- `access_key_id`
- `secret_access_key`
- `session_token`
2019-10-31 23:06:26 -04:00
2019-01-23 15:48:38 -05:00
## Security
2020-09-21 23:09:49 -04:00
### Server Side Request Forgery (SSRF) attacks
2019-01-23 15:48:38 -05:00
As URLs for Kubernetes clusters are user controlled it is easily
susceptible to Server Side Request Forgery (SSRF) attacks. You should
understand the mitigation strategies if you are adding more API calls to
a cluster.
Mitigation strategies include:
1. Not allowing redirects to attacker controller resources:
2019-09-18 14:06:14 -04:00
[`Kubeclient::KubeClient` ](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb# )
2019-01-23 15:48:38 -05:00
can be configured to disallow any redirects by passing in
`http_max_redirects: 0` as an option.
1. Not exposing error messages: by doing so, we
prevent attackers from triggering errors to expose results from
attacker controlled requests. For example, we do not expose (or store)
raw error messages:
```ruby
rescue Kubernetes::HttpError => e
# bad
# app.make_errored!("Kubernetes error: #{e.message}")
# good
app.make_errored!("Kubernetes error: #{e.error_code}")
```
2020-09-21 23:09:49 -04:00
## Debugging Kubernetes integrations
2019-01-23 15:48:38 -05:00
Logs related to the Kubernetes integration can be found in
2019-08-27 04:44:07 -04:00
[`kubernetes.log` ](../administration/logs.md#kuberneteslog ). On a local
2020-09-21 23:09:49 -04:00
GDK install, these logs are present in `log/kubernetes.log` .
2019-01-23 15:48:38 -05:00
Some services such as
2019-09-18 14:06:14 -04:00
[`Clusters::Applications::InstallService` ](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/clusters/applications/install_service.rb#L18 )
2019-01-23 15:48:38 -05:00
rescues `StandardError` which can make it harder to debug issues in an
development environment. The current workaround is to temporarily
comment out the `rescue` in your local development source.
2020-03-10 08:08:16 -04:00
You can also follow the installation logs to debug issues related to
2019-01-23 15:48:38 -05:00
installation. Once the installation/upgrade is underway, wait for the
pod to be created. Then run the following to obtain the pods logs as
they are written:
2020-01-30 10:09:15 -05:00
```shell
2019-01-23 15:48:38 -05:00
kubectl logs < pod_name > --follow -n gitlab-managed-apps
```
2019-10-30 23:06:45 -04:00
## GitLab Managed Apps
2020-09-21 23:09:49 -04:00
GitLab provides [GitLab Managed Apps ](../user/clusters/applications.md ), a one-click
install for various applications which can be added directly to your configured cluster.
2019-10-30 23:06:45 -04:00
**< i class = "fa fa-youtube-play youtube" aria-hidden = "true" > < / i >
2020-09-21 23:09:49 -04:00
For an overview of how to add a new GitLab-managed app, see
[How to add GitLab-managed-apps to Kubernetes integration ](https://youtu.be/mKm-jkranEk ).**