gitlab-org--gitlab-foss/doc/administration/troubleshooting/kubernetes_cheat_sheet.md

type
reference

Kubernetes, GitLab and You

This is a list of useful information regarding Kubernetes that the GitLab Support Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone can make use of the Support team's collected knowledge

CAUTION: Caution: These commands can alter or break your Kubernetes components so use these at your own risk.

If you are on a paid tier and are not sure how to use these commands, it is best to contact Support and they will assist you with any issues you are having.

Generic kubernetes commands

• How to authorize to your GCP project (can be especially useful if you have projects under different GCP accounts):

  gcloud auth login
  

• How to access Kubernetes dashboard:

  # for minikube:
  minikube dashboard —url
  # for non-local installations if access via Kubectl is configured:
  kubectl proxy
  

• How to ssh to a Kubernetes node and enter the container as root https://github.com/kubernetes/kubernetes/issues/30656:

  • For GCP, you may find the node name and run gcloud compute ssh node-name.
  • List containers using docker ps.
  • Enter container using docker exec --user root -ti container-id bash.

• How to copy a file from local machine to a pod:

  kubectl cp file-name pod-name:./destination-path
  

• What to do with pods in CrashLoopBackoff status:

  • Check logs via Kubernetes dashboard.

  • Check logs via Kubectl:

    kubectl logs <unicorn pod> -c dependencies
    

• How to tail all Kubernetes cluster events in real time:

  kubectl get events -w --all-namespaces
  

• How to get logs of the previously terminated pod instance:

  kubectl logs <pod-name> --previous
  

  NOTE: Note: No logs are kept in the containers/pods themselves, everything is written to stdout. This is the principle of Kubernetes, read Twelve-factor app for details.

GitLab-specific kubernetes information

• Minimal config that can be used to test a Kubernetes helm chart can be found here.

• Tailing logs of a separate pod. An example for a unicorn pod:

  kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs -c unicorn
  

• Tail and follow all pods that share a label (in this case, unicorn):

  # all containers in the unicorn pods
  kubectl logs -f -l app=unicorn --all-containers=true --max-log-requests=50
  
  # only the unicorn containers in all unicorn pods
  kubectl logs -f -l app=unicorn -c unicorn --max-log-requests=50
  

• One can stream logs from all containers at once, similar to the Omnibus command gitlab-ctl tail:

  kubectl logs -f -l release=gitlab --all-containers=true --max-log-requests=100
  

• Check all events in the gitlab namespace (the namespace name can be different if you specified a different one when deploying the helm chart):

  kubectl get events -w --namespace=gitlab
  

• Most of the useful GitLab tools (console, rake tasks, etc) are found in the task-runner pod. You may enter it and run commands inside or run them from the outside:

  # find the pod
  kubectl get pods | grep task-runner
  
  # enter it
  kubectl exec -it <task-runner-pod-name> bash
  
  # open rails console
  # rails console can be also called from other GitLab pods
  /srv/gitlab/bin/rails console
  
  # source-style commands should also work
  /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production
  
  # run GitLab check. Note that the output can be confusing and invalid because of the specific structure of GitLab installed via helm chart
  /usr/local/bin/gitlab-rake gitlab:check
  
  # open console without entering pod
  kubectl exec -it <task-runner-pod-name> /srv/gitlab/bin/rails console
  
  # check the status of DB migrations
  kubectl exec -it <task-runner-pod-name> /usr/local/bin/gitlab-rake db:migrate:status
  

  You can also use gitlab-rake, instead of /usr/local/bin/gitlab-rake.

• Troubleshooting Operations > Kubernetes integration:

  • Check the output of kubectl get events -w --all-namespaces.
  • Check the logs of pods within gitlab-managed-apps namespace.
  • On the side of GitLab check sidekiq log and kubernetes log. When GitLab is installed via Helm Chart, kubernetes.log can be found inside the sidekiq pod.

• How to get your initial admin password https://docs.gitlab.com/charts/installation/deployment.html#initial-login:

  # find the name of the secret containing the password
  kubectl get secrets | grep initial-root
  # decode it
  kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo
  

• How to connect to a GitLab Postgres database:

  kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole -p
  

• How to get info about Helm installation status:

  helm status name-of-installation
  

• How to update GitLab installed using Helm Chart:

  helm repo upgrade
  
  # get current values and redirect them to yaml file (analogue of gitlab.rb values)
  helm get values <release name> > gitlab.yaml
  
  # run upgrade itself
  helm upgrade <release name> <chart path> -f gitlab.yaml
  

  After https://canary.gitlab.com/charts/gitlab/issues/780 is fixed, it should be possible to use Updating GitLab using the Helm Chart for upgrades.

• How to apply changes to GitLab config:

  • Modify the gitlab.yaml file.

  • Run the following command to apply changes:

    helm upgrade <release name> <chart path> -f gitlab.yaml
    

Installation of minimal GitLab config via Minukube on macOS

This section is based on Developing for Kubernetes with Minikube and Helm. Refer to those documents for details.

• Install Kubectl via Homebrew:

  brew install kubernetes-cli
  

• Install Minikube via Homebrew:

  brew cask install minikube
  

• Start Minikube and configure it. If Minikube cannot start, try running minikube delete && minikube start and repeat the steps:

  minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work
  minikube addons enable ingress
  minikube addons enable kube-dns
  

• Install Helm via Homebrew and initialize it:

  brew install kubernetes-helm
  helm init --service-account tiller
  

• Copy the file https://gitlab.com/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml to your workstation.

• Find the IP address in the output of minikube ip and update the yaml file with this IP address.

• Install the GitLab Helm Chart:

  helm repo add gitlab https://charts.gitlab.io
  helm install --name gitlab -f <path-to-yaml-file> gitlab/gitlab
  

  If you want to modify some GitLab settings, you can use the above-mentioned config as a base and create your own yaml file.

• Monitor the installation progress via helm status gitlab and minikube dashboard. The installation could take up to 20-30 minutes depending on the amount of resources on your workstation.

• When all the pods show either a Running or Completed status, get the GitLab password as described in Initial login, and log in to GitLab via the UI. It will be accessible via https://gitlab.domain where domain is the value provided in the yaml file.