From 54d3ad5217da41f657ef714ec889bb29de922649 Mon Sep 17 00:00:00 2001 From: Thong Kuah Date: Thu, 24 Jan 2019 09:48:38 +1300 Subject: [PATCH] Development guide for Kubernetes Start writing some development guidelines in relation to our Kubernetes integration --- doc/development/README.md | 1 + doc/development/kubernetes.md | 126 ++++++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+) create mode 100644 doc/development/kubernetes.md diff --git a/doc/development/README.md b/doc/development/README.md index d5829e31343..13646cbfe48 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -49,6 +49,7 @@ description: 'Learn how to contribute to GitLab.' - [Working with the GitHub importer](github_importer.md) - [Import/Export development documentation](import_export.md) - [Working with Merge Request diffs](diffs.md) +- [Kubernetes integration guidelines](kubernetes.md) - [Permissions](permissions.md) - [Prometheus metrics](prometheus_metrics.md) - [Guidelines for reusing abstractions](reusing_abstractions.md) diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md new file mode 100644 index 00000000000..4b2d48903ac --- /dev/null +++ b/doc/development/kubernetes.md @@ -0,0 +1,126 @@ +# Kubernetes integration - development guidelines + +This document provides various guidelines when developing for GitLab's +[Kubernetes integration](../user/project/clusters/index.md). + +## Development + +### Architecture + +Some Kubernetes operations, such as creating restricted project +namespaces are performed on the GitLab Rails application. These +operations are performed using a [client library](#client-library). +These operations will carry an element of risk as the operations will be +run as the same user running the GitLab Rails application, see the +[security](#security) section below. + +Some Kubernetes operations, such as installing cluster applications are +performed on one-off pods on the Kubernetes cluster itself. These +installation pods are currently named `install-` and +are created within the `gitlab-managed-apps` namespace. + +In terms of code organization, we generally add objects that represent +Kubernetes resources in +[`lib/gitlab/kubernetes`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/kubernetes). + +### Client library + +We use the [`kubeclient`](https://rubygems.org/gems/kubeclient) gem to +perform Kubernetes API calls. As the `kubeclient` gem does not support +different API Groups (e.g. `apis/rbac.authorization.k8s.io`) from a +single client, we have created a wrapper class, +[`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/kubernetes/kube_client.rb) +that will enable you to achieve this. + +Selected Kubernetes API groups are currently supported. Do add support +for new API groups or methods to +[`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/kubernetes/kube_client.rb) +if you need to use them. New API groups or API group versions can be +added to `SUPPORTED_API_GROUPS` - internally, this will create an +internal client for that group. New methods can be added as a delegation +to the relevant internal client. + +### Performance considerations + +All calls to the Kubernetes API must be in a background process. Do not +perform Kubernetes API calls within a web request as this will block +unicorn and can easily lead to a Denial Of Service (DoS) attack in GitLab as +the Kubernetes cluster response times are outside of our control. + +The easiest way to ensure your calls happen a background process is to +delegate any such work to happen in a [sidekiq +worker](sidekiq_style_guide.md). + +There are instances where you would like to make calls to Kubernetes and +return the response and as such a background worker does not seem to be +a good fit. For such cases you should make use of [reactive +caching](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/models/concerns/reactive_caching.rb). +For example: + +```ruby + def calculate_reactive_cache! + { pods: cluster.platform_kubernetes.kubeclient.get_pods } + end + + def pods + with_reactive_cache do |data| + data[:pods] + end + end +``` + +### Testing + +We have some Webmock stubs in +[`KubernetesHelpers`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/support/helpers/kubernetes_helpers.rb) +which can help with mocking out calls to Kubernetes API in your tests. + +## Security + +### SSRF + +As URLs for Kubernetes clusters are user controlled it is easily +susceptible to Server Side Request Forgery (SSRF) attacks. You should +understand the mitigation strategies if you are adding more API calls to +a cluster. + +Mitigation strategies include: + +1. Not allowing redirects to attacker controller resources: + [`Kubeclient::KubeClient`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/kubernetes/kube_client.rb#) + can be configured to disallow any redirects by passing in + `http_max_redirects: 0` as an option. +1. Not exposing error messages: by doing so, we + prevent attackers from triggering errors to expose results from + attacker controlled requests. For example, we do not expose (or store) + raw error messages: + + ```ruby + rescue Kubernetes::HttpError => e + # bad + # app.make_errored!("Kubernetes error: #{e.message}") + + # good + app.make_errored!("Kubernetes error: #{e.error_code}") + ``` + +## Debugging + +Logs related to the Kubernetes integration can be found in +[kubernetes.log](../administration/logs.md#kuberneteslog). On a local +GDK install, this will be present in `log/kubernetes.log`. + +Some services such as +[`Clusters::Applications::InstallService`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/services/clusters/applications/install_service.rb#L18) +rescues `StandardError` which can make it harder to debug issues in an +development environment. The current workaround is to temporarily +comment out the `rescue` in your local development source. + +You can also follow the installation pod logs to debug issues related to +installation. Once the installation/upgrade is underway, wait for the +pod to be created. Then run the following to obtain the pods logs as +they are written: + +```bash +kubectl logs --follow -n gitlab-managed-apps +```