19 KiB

Raw Blame History

stage	group	info
Create	Source Code	To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments

FIPS compliance

FIPS is short for "Federal Information Processing Standard", a document which defines certain security practices for a "cryptographic module" (CM). It aims to ensure a certain security floor is met by vendors selling products to U.S. Federal institutions.

WARNING: GitLab is not FIPS compliant, even when built and run on a FIPS-enforcing system. Large parts of the build are broken, and many features use forbidden cryptographic primitives. Running GitLab on a FIPS-enforcing system is not supported and may result in data loss. This document is intended to help engineers looking to develop FIPS-related fixes. It is not intended to be used to run a production GitLab instance.

There are two current FIPS standards: 140-2 and 140-3. At GitLab we usually mean FIPS 140-2.

Current status

GitLab Inc has not committed to making GitLab FIPS-compliant at this time. We are performing initial investigations to see how much work such an effort would be.

Read Epic &5104 for more information on the status of the investigation.

FIPS compliance at GitLab

In a FIPS context, compliance is a form of self-certification - if we say we are "FIPS compliant", we mean that we believe we are. There are no external certifications to acquire, but if we are aware of non-compliant areas in GitLab, we cannot self-certify in good faith.

The known areas of non-compliance are tracked in Epic &5104.

To be compliant, all components (GitLab itself, Gitaly, etc) must be compliant, along with the communication between those components, and any storage used by them. Where functionality cannot be brought into compliance, it must be disabled when FIPS mode is enabled.

FIPS validation at GitLab

Unlike FIPS compliance, FIPS validation is a formal declaration of compliance by an accredited auditor. The requirements needed to pass the audit are the same as for FIPS compliance.

A list of FIPS-validated modules can be found at the NIST (National Institute of Standards and Technology) cryptographic module validation program.

Setting up a FIPS-enabled development environment

The simplest approach is to set up a virtual machine running Red Hat Enterprise Linux 8.

Red Hat provide free licenses to developers, and permit the CD image to be downloaded from the Red Hat developer's portal. Registration is required.

After the virtual machine is set up, you can follow the GDK installation instructions, including the advanced instructions for RHEL. Note that asdf is not used for dependency management because it's essential to use the RedHat-provided Go compiler and other system dependencies.

Enable FIPS mode

After GDK and its dependencies are installed, run this command (as root) and restart the virtual machine:

fips-mode-setup --enable

You can check whether it's taken effect by running:

fips-mode-setup --check

In this environment, OpenSSL refuses to perform cryptographic operations forbidden by the FIPS standards. This enables you to reproduce FIPS-related bugs, and validate fixes.

You should be able to open a web browser inside the virtual machine and log in to the GitLab instance.

You can disable FIPS mode again by running this command, then restarting the virtual machine:

fips-mode-setup --disable

Detect FIPS enablement in code

You can query Gitlab::FIPS in Ruby code to determine if the instance is FIPS-enabled:

def default_min_key_size(name)
  if Gitlab::FIPS.enabled?
    Gitlab::SSHPublicKey.supported_sizes(name).select(&:positive?).min || -1
  else
    0
  end
end

Nightly Omnibus FIPS builds

The Distribution team has created nightly FIPS Omnibus builds. These GitLab builds are compiled to use the system OpenSSL instead of the Omnibus-embedded version of OpenSSL.

See the section on how FIPS builds are created.

Runner

See the documentation on installing a FIPS-compliant GitLab Runner.

Set up a FIPS-enabled cluster

You can use the GitLab Environment Toolkit to spin up a FIPS-enabled cluster for development and testing. These instructions use Amazon Web Services (AWS) because that is the first target environment, but you can adapt them for other providers.

Set up your environment

To get started, your AWS account must subscribe to a FIPS-enabled Amazon Machine Image (AMI) in the AWS Marketplace console.

This example assumes that the Ubuntu Pro 20.04 FIPS LTS AMI by Canonical Group Limited has been added your account. This operating system is used for virtual machines running in Amazon EC2.

Omnibus

The simplest way to get a FIPS-enabled GitLab cluster is to use an Omnibus reference architecture. See the GET Quick Start Guide for more details. The following instructions build on the Quick Start and are also necessary for Cloud Native Hybrid installations.

Terraform: Use a FIPS AMI

Follow the guide to set up Terraform and Ansible.

After step 2b, create a data.tf in your environment (for example, gitlab-environment-toolkit/terraform/environments/gitlab-10k/inventory/data.tf):

data "aws_ami" "ubuntu_20_04_fips" {
  count = 1

  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu-pro-fips-server/images/hvm-ssd/ubuntu-focal-20.04-amd64-pro-fips-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["aws-marketplace"]
}

Add the custom ami_id to use this AMI in environment.tf. For example, in gitlab-environment-toolkit/terraform/environments/gitlab-10k/inventory/environment.tf:

module "gitlab_ref_arch_aws" {
  source = "../../modules/gitlab_ref_arch_aws"

  prefix = var.prefix
  ami_id = data.aws_ami.ubuntu_20_04_fips[0].id
  ...

NOTE: GET does not allow the AMI to change on EC2 instances after it has been deployed via terraform apply. Since an AMI change would tear down an instance, this would result in data loss: not only would disks be destroyed, but also GitLab secrets would be lost. There is a Terraform lifecycle rule to ignore AMI changes.

Ansible: Specify the FIPS Omnibus builds

The standard Omnibus GitLab releases build their own OpenSSL library, which is not FIPS-validated. However, we have nightly builds that create Omnibus packages that link against the operating system's OpenSSL library. To use this package, update the gitlab_edition and gitlab_repo_script_url fields in the Ansible vars.yml. For example, you might modify gitlab-environment-toolkit/ansible/environments/gitlab-10k/inventory/vars.yml in this way:

all:
  vars:
    ...
    gitlab_repo_script_url: "https://packages.gitlab.com/install/repositories/gitlab/nightly-fips-builds/script.deb.sh"
    gitlab_edition: "gitlab-fips"

Cloud Native Hybrid

A Cloud Native Hybrid install uses both Omnibus and Cloud Native GitLab (CNG) images. The previous instructions cover the Omnibus part, but two additional steps are needed to enable FIPS in CNG:

Use a custom Amazon Elastic Kubernetes Service (EKS) AMI.
Use GitLab containers built with RedHat's Universal Base Image (UBI).

Build a custom EKS AMI

Because Amazon does not yet publish a FIPS-enabled AMI, you have to build one yourself with Packer.

Amazon publishes the following Git repositories with information about custom EKS AMIs:

This GitHub pull request makes it possible to create an Amazon Linux 2 EKS AMI with FIPS enabled for Kubernetes v1.21. To build an image:

Install Packer.

Run the following:

git clone https://github.com/awslabs/amazon-eks-ami
cd amazon-eks-ami
git fetch origin pull/898/head:fips-ami
git checkout fips-ami
AWS_DEFAULT_REGION=us-east-1 make 1.21-fips # Be sure to set the region accordingly

If you are using a different version of Kubernetes, adjust the make command and Makefile accordingly.

When the AMI build is done, a new AMI should be created with a message such as the following:

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
us-west-2: ami-0a25e760cd00b027e

In this example, the AMI ID is ami-0a25e760cd00b027e, but your value may be different.

Building a RHEL-based system with FIPS enabled should be possible, but there is an outstanding issue preventing the Packer build from completing.

Terraform: Use a custom EKS AMI

Now you can set the custom EKS AMI.

In environment.tf, add eks_ami_id = var.eks_ami_id so you can pass this variable to the AWS reference architecture module. For example, in gitlab-environment-toolkit/terraform/environments/gitlab-10k/inventory/environment.tf:
```
module "gitlab_ref_arch_aws" {
  source = "../../modules/gitlab_ref_arch_aws"

  prefix = var.prefix
  ami_id = data.aws_ami.ubuntu_20_04_fips[0].id
  eks_ami_id = var.eks_ami_id
  ....
```

In variables.tf, define a eks_ami_id with the AMI ID in the previous step:

variable "eks_ami_id" {
  default = "ami-0a25e760cd00b027e"
}

Ansible: Use UBI images

CNG uses a Helm Chart to manage which container images to deploy. By default, GET deploys the latest released versions that use Debian-based containers.

To switch to UBI-based containers, edit the Ansible vars.yml to use custom Charts variables:

all:
  vars:
    ...
    gitlab_charts_custom_config_file: '/path/to/gitlab-environment-toolkit/ansible/environments/gitlab-10k/inventory/charts.yml'

Now create charts.yml in the location specified above and specify tags with a -fips suffix. For example:

global:
  image:
    pullPolicy: Always
  certificates:
    image:
      tag: master-fips
  kubectl:
    image:
      tag: master-fips

gitlab:
  gitaly:
    image:
      tag: master-fips
  gitlab-exporter:
    image:
      tag: master-fips
  gitlab-shell:
    image:
      tag: main-fips # The default branch is main, not master
  gitlab-mailroom:
    image:
      tag: master-fips
  migrations:
    image:
      tag: master-fips
  sidekiq:
    image:
      tag: master-fips
  toolbox:
    image:
      tag: master-fips
  webservice:
    image:
      tag: master-fips
    workhorse:
      tag: master-fips

nginx-ingress:
  controller:
    image:
      repository: registry.gitlab.com/stanhu/gitlab-test-images/k8s-staging-ingress-nginx/controller
      tag: v1.2.0-beta.1
      pullPolicy: Always
      digest: sha256:ace38833689ad34db4a46bc1e099242696eb800def88f02200a8615530734116

The above example shows a FIPS-enabled nginx-ingress image. See this issue for more details on how to build NGINX and the Ingress Controller.

You can also use release tags, but the versioning is tricky because each component may use its own versioning scheme. For example, for GitLab v15.1:

global:
  certificates:
    image:
      tag: 20211220-r0-fips
  kubectl:
    image:
      tag: 1.18.20-fips

gitlab:
  gitaly:
    image:
      tag: v15.1.0-fips
  gitlab-exporter:
    image:
      tag: 11.15.2-fips
  gitlab-shell:
    image:
      tag: v15.1.0-fips
  gitlab-mailroom:
    image:
      tag: v15.1.0-fips
  migrations:
    image:
      tag: v15.1.0-fips
  sidekiq:
    image:
      tag: v15.1.0-fips
  toolbox:
    image:
      tag: v15.1.0-fips
  webservice:
    image:
      tag: v15.1.0-fips
    workhorse:
      tag: v15.1.0-fips

Verify FIPS

The following sections describe ways you can verify if FIPS is enabled.

Kernel

$ cat /proc/sys/crypto/fips_enabled
1

Ruby (Omnibus images)

$ /opt/gitlab/embedded/bin/irb
irb(main):001:0> require 'openssl'; OpenSSL.fips_mode
=> true

Ruby (CNG images)

$ irb
irb(main):001:0> require 'openssl'; OpenSSL.fips_mode
=> true

Go

Google maintains a dev.boringcrypto branch in the Golang compiler that makes it possible to statically link BoringSSL, a FIPS-validated module forked from OpenSSL. However, BoringSSL is not intended for public use.

We use golang-fips, a fork of the dev.boringcrypto branch to build Go programs that dynamically link OpenSSL via dlopen. This has several advantages:

Using a FIPS-validated, system OpenSSL is straightforward.
This is the source code used by Red Hat's go-toolset package.
Unlike go-toolset, this fork appears to keep up with the latest Go releases.

However, cgo must be enabled via CGO_ENABLED=1 for this to work. There is a performance hit when calling into C code.

Projects that are compiled with golang-fips on Linux x86 automatically get built the crypto routines that use OpenSSL. While the boringcrypto build tag is automatically present, no extra build tags are actually needed. There are specific build tags that disable these crypto hooks.

We can check whether a given binary is using OpenSSL via go tool nm and look for symbols named Cfunc__goboringcrypto. For example:

$ go tool nm nginx-ingress-controller  | grep Cfunc__goboringcrypto | tail
 2a0b650 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA384_Final
 2a0b658 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA384_Init
 2a0b660 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA384_Update
 2a0b668 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA512_Final
 2a0b670 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA512_Init
 2a0b678 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA512_Update
 2a0b680 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ECDSA_sign
 2a0b688 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ECDSA_verify
 2a0b690 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ERR_error_string_n
 2a0b698 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ERR_get_error

In addition, LabKit contains routines to check whether FIPS is enabled.

How FIPS builds are created

Many GitLab projects (for example: Gitaly, GitLab Pages) have standardized on using FIPS_MODE=1 make to build FIPS binaries locally.

Omnibus

The Omnibus FIPS builds are triggered with the USE_SYSTEM_SSL environment variable set to true. When this environment variable is set, the Omnibus recipes dependencies such as curl, NGINX, and libgit2 will link against the system OpenSSL. OpenSSL will NOT be included in the Omnibus build.

The Omnibus builds are created using container images that use the golang-fips compiler. For example, this job created the registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/centos_8_fips:3.3.1 image used to build packages for RHEL 8.

Add a new FIPS build for another Linux distribution

First, you need to make sure there is an Omnibus builder image for the desired Linux distribution. The images used to build Omnibus packages are created with Omnibus Builder images.

Review this merge request. A new image can be added by:

Adding CI jobs with the _fips suffix (for example: ubuntu_18.04_fips).
Making sure the Dockerfile uses Snippets.new(fips: fips).populate instead of Snippets.new.populate.

After this image has been tagged, add a new CI job to Omnibus GitLab.

Cloud Native GitLab (CNG)

The Cloud Native GitLab CI pipeline generates images using several base images:

Debian
Red Hat's Universal Base Image (UBI)

UBI images ship with the same OpenSSL package as those used by RHEL. This makes it possible to build FIPS-compliant binaries without needing RHEL. Note that RHEL 8.2 ships a FIPS-validated OpenSSL, but 8.5 is in review for FIPS validation.

This merge request introduces a FIPS pipeline for CNG images. Images tagged for FIPS have the -fips suffix. For example, the webservice container has the following tags:

master
master-ubi8
master-fips

Testing merge requests with a FIPS pipeline

Merge requests that can trigger Package and QA, can trigger a FIPS package and a Reference Architecture test pipeline. The base image used for the trigger is Ubuntu 20.04 FIPS:

Trigger package-and-qa, if not already triggered.
On the gitlab-omnibus-mirror child pipeline, manually trigger Trigger:package:fips.
When the package job is complete, manually trigger the RAT:FIPS job.

19 KiB Raw Blame History

FIPS compliance

Current status

FIPS compliance at GitLab

FIPS validation at GitLab

Setting up a FIPS-enabled development environment

Enable FIPS mode

Detect FIPS enablement in code

Nightly Omnibus FIPS builds

Runner

Set up a FIPS-enabled cluster

Set up your environment

Omnibus

Terraform: Use a FIPS AMI

Ansible: Specify the FIPS Omnibus builds

Cloud Native Hybrid

Build a custom EKS AMI

Terraform: Use a custom EKS AMI

Ansible: Use UBI images

Verify FIPS

Kernel

Ruby (Omnibus images)

Ruby (CNG images)

Go

How FIPS builds are created

Omnibus

Add a new FIPS build for another Linux distribution

Cloud Native GitLab (CNG)

Testing merge requests with a FIPS pipeline

19 KiB

Raw Blame History