kotovalexarian-likes-gitlab/gitlab-org--gitlab-foss
1
0
Fork 0
Code Releases Activity

Add latest changes from gitlab-org/gitlab@master

Browse source
This commit is contained in:
GitLab Bot 2019-10-24 18:07:05 +00:00
parent 8db8e2a342
commit c4edbefa45
9 changed files with 453 additions and 32 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.gitlab-ci.yml
View file

@ -1,6 +1,7 @@
image: "registry.gitlab.com/gitlab-org/gitlab-build-images:ruby-2.6.3-golang-1.11-git-2.22-chrome-73.0-node-12.x-yarn-1.16-postgresql-9.6-graphicsmagick-1.3.33"
stages:
- sync
- prepare
- quick-test
- test
@ -40,3 +41,4 @@ include:
- local: .gitlab/ci/setup.gitlab-ci.yml
- local: .gitlab/ci/test-metadata.gitlab-ci.yml
- local: .gitlab/ci/yaml.gitlab-ci.yml
- local: .gitlab/ci/releases.gitlab-ci.yml

22
.gitlab/ci/releases.gitlab-ci.yml Normal file
View file

@ -0,0 +1,22 @@
---
# Syncs any changes pushed to a stable branch to the corresponding CE stable
# branch. We run this prior to any tests so that random failures don't prevent a
# sync.
sync-stable-branch:
# We don't need/want any global before/after commands, so we overwrite these
# settings.
image: alpine:edge
stage: sync
# This job should only run on EE stable branches on the canonical GitLab.com
# repository.
only:
variables:
- $CI_SERVER_HOST == "gitlab.com"
refs:
- /^[\d-]+-stable-ee$/@gitlab-org/gitlab
before_script:
- apk add --no-cache --update curl bash
after_script: []
script:
- bash scripts/sync-stable-branch.sh

155
doc/development/documentation/site_architecture/index.md
View file

@ -4,14 +4,16 @@ description: "Learn how GitLab's documentation website is architectured."
# Documentation site architecture
Learn how we build and architecture [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs)
and deploy it to <https://docs.gitlab.com>.
The [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) project hosts
the repository which is used to generate the GitLab documentation website and
is deployed to <https://docs.gitlab.com>. It uses the [Nanoc](http://nanoc.ws)
static site generator.
## Repository
## Architecture
While the source of the documentation content is stored in GitLab's respective product
repositories, the source that is used to build the documentation site _from that content_
is located at <https://gitlab.com/gitlab-org/gitlab-docs>.
repositories, the source that is used to build the documentation
site _from that content_ is located at <https://gitlab.com/gitlab-org/gitlab-docs>.
The following diagram illustrates the relationship between the repositories
from where content is sourced, the `gitlab-docs` project, and the published output.
@ -43,8 +45,23 @@ from where content is sourced, the `gitlab-docs` project, and the published outp
G --> L
```
See the [README there](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/README.md)
for detailed information.
You will not find any GitLab docs content in the `gitlab-docs` repository.
All documentation files are hosted in the respective repository of each
product, and all together are pulled to generate the docs website:
- [GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master/doc)
- [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc)
- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/tree/master/docs)
- [GitLab Chart](https://gitlab.com/charts/gitlab/tree/master/doc)
NOTE: **Note:**
In September 2019, we [moved towards a single codebase](https://gitlab.com/gitlab-org/gitlab-ee/issues/2952),
as such the docs for CE and EE are now identical. For historical reasons and
in order not to break any existing links throughout the internet, we still
maintain the CE docs (`https://docs.gitlab.com/ce/`), although it is hidden
from the website, and is now a symlink to the EE docs. When
[Pages supports redirects](https://gitlab.com/gitlab-org/gitlab-pages/issues/24),
we will be able to remove this completely.
## Assets
@ -73,28 +90,112 @@ Read through [the global navigation documentation](global_nav.md) to understand:
- How the global navigation is built.
- How to add new navigation items.
## Deployment
The docs site is deployed to production with GitLab Pages, and previewed in
merge requests with Review Apps.
The deployment aspects will be soon transferred from the [original document](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/README.md)
to this page.
<!--
## Repositories
TBA
## Search engine
TBA
## Versions
TBA
## Helpers
TBA
-->
## Using YAML data files
The easiest way to achieve something similar to
[Jekyll's data files](https://jekyllrb.com/docs/datafiles/) in Nanoc is by
using the [`@items`](https://nanoc.ws/doc/reference/variables/#items-and-layouts)
variable.
The data file must be placed inside the `content/` directory and then it can
be referenced in an ERB template.
Suppose we have the `content/_data/versions.yaml` file with the content:
```yaml
versions:
- 10.6
- 10.5
- 10.4
```
We can then loop over the `versions` array with something like:
```erb
<% @items['/_data/versions.yaml'][:versions].each do | version | %>
<h3><%= version %></h3>
<% end &>
```
Note that the data file must have the `yaml` extension (not `yml`) and that
we reference the array with a symbol (`:versions`).
## Bumping versions of CSS and Javascript
Whenever the custom CSS and Javascript files under `content/assets/` change,
make sure to bump their version in the frontmatter. This method guarantees that
your changes will take effect by clearing the cache of previous files.
Always use Nanoc's way of including those files, do not hardcode them in the
layouts. For example use:
```erb
<script async type="application/javascript" src="<%= @items['/assets/javascripts/badges.*'].path %>"></script>
<link rel="stylesheet" href="<%= @items['/assets/stylesheets/toc.*'].path %>">
```
The links pointing to the files should be similar to:
```erb
<%= @items['/path/to/assets/file.*'].path %>
```
Nanoc will then build and render those links correctly according with what's
defined in [`Rules`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/Rules).
## Linking to source files
A helper called [`edit_on_gitlab`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/lib/helpers/edit_on_gitlab.rb) can be used
to link to a page's source file. We can link to both the simple editor and the
web IDE. Here's how you can use it in a Nanoc layout:
- Default editor: `<a href="<%= edit_on_gitlab(@item, editor: :simple) %>">Simple editor</a>`
- Web IDE: `<a href="<%= edit_on_gitlab(@item, editor: :webide) %>">Web IDE</a>`
If you don't specify `editor:`, the simple one is used by default.
## Algolia search engine
The docs site uses [Algolia docsearch](https://community.algolia.com/docsearch/)
for its search function. This is how it works:
1. GitLab is a member of the [docsearch program](https://community.algolia.com/docsearch/#join-docsearch-program),
which is the free tier of [Algolia](https://www.algolia.com/).
1. Algolia hosts a [doscsearch config](https://github.com/algolia/docsearch-configs/blob/master/configs/gitlab.json)
for the GitLab docs site, and we've worked together to refine it.
1. That [config](https://community.algolia.com/docsearch/config-file.html) is
parsed by their [crawler](https://community.algolia.com/docsearch/crawler-overview.html)
every 24h and [stores](https://community.algolia.com/docsearch/inside-the-engine.html)
the [docsearch index](https://community.algolia.com/docsearch/how-do-we-build-an-index.html)
on [Algolia's servers](https://community.algolia.com/docsearch/faq.html#where-is-my-data-hosted%3F).
1. On the docs side, we use a [docsearch layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/docsearch.html) which
is present on pretty much every page except <https://docs.gitlab.com/search/>,
which uses its [own layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/instantsearch.html). In those layouts,
there's a javascript snippet which initiates docsearch by using an API key
and an index name (`gitlab`) that are needed for Algolia to show the results.
NOTE: **For GitLab employees:**
The credentials to access the Algolia dashboard are stored in 1Password. If you
want to receive weekly reports of the search usage, search the Google doc with
title "Email, Slack, and GitLab Groups and Aliases", search for `docsearch`,
and add a comment with your email to be added to the alias that gets the weekly
reports.
## Monthly release process (versions)
The docs website supports versions and each month we add the latest one to the list.
For more information, read about the [monthly release process](release_process.md).
## Review Apps for documentation merge requests
If you are contributing to GitLab docs read how to [create a Review App with each
merge request](../index.md#previewing-the-changes-live).

241
doc/development/documentation/site_architecture/release_process.md Normal file
View file

@ -0,0 +1,241 @@
# GitLab Docs monthly release process
The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/)
contains all needed Dockerfiles to build and deploy the versioned website. It
is heavily inspired by Docker's
[Dockerfile](https://github.com/docker/docker.github.io/blob/06ed03db13895bfe867761b6fc2ad40acf6026dd/Dockerfile).
The following Dockerfiles are used.
| Dockerfile | Docker image | Description |
| ---------- | ------------ | ----------- |
| [`Dockerfile.bootstrap`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.bootstrap) | `gitlab-docs:bootstrap` | Contains all the dependencies that are needed to build the website. If the gems are updated and `Gemfile{,.lock}` changes, the image must be rebuilt. |
| [`Dockerfile.builder.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.builder.onbuild) | `gitlab-docs:builder-onbuild` | Base image to build the docs website. It uses `ONBUILD` to perform all steps and depends on `gitlab-docs:bootstrap`. |
| [`Dockerfile.nginx.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.nginx.onbuild) | `gitlab-docs:nginx-onbuild` | Base image to use for building documentation archives. It uses `ONBUILD` to perform all required steps to copy the archive, and relies upon its parent `Dockerfile.builder.onbuild` that is invoked when building single documentation achives (see the `Dockerfile` of each branch. |
| [`Dockerfile.archives`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.archives) | `gitlab-docs:archives` | Contains all the versions of the website in one archive. It copies all generated HTML files from every version in one location. |
## How to build the images
Although build images are built automatically via GitLab CI/CD, you can build
and tag all tooling images locally:
1. Make sure you have [Docker installed](https://docs.docker.com/install/).
1. Make sure you're on the `dockerfiles/` directory of the `gitlab-docs` repo.
1. Build the images:
```sh
docker build -t registry.gitlab.com/gitlab-org/gitlab-docs:bootstrap -f Dockerfile.bootstrap ../
docker build -t registry.gitlab.com/gitlab-org/gitlab-docs:builder-onbuild -f Dockerfile.builder.onbuild ../
docker build -t registry.gitlab.com/gitlab-org/gitlab-docs:nginx-onbuild -f Dockerfile.nginx.onbuild ../
```
For each image, there's a manual job under the `images` stage in
[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/.gitlab-ci.yml) which can be invoked at will.
## Monthly release process
When a new GitLab version is released on the 22nd, we need to create the respective
single Docker image, and update some files so that the dropdown works correctly.
### 1. Add the chart version
Since the charts use a different version number than all the other GitLab
products, we need to add a
[version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html):
1. Check that there is a [stable branch created](https://gitlab.com/gitlab-org/charts/gitlab/-/branches)
for the new chart version. If you're unsure or can't find it, drop a line in
the `#g_delivery` channel.
1. Make sure you're on the root path of the `gitlab-docs` repo.
1. Open `content/_data/chart_versions.yaml` and add the new stable branch version using the
version mapping. Note that only the `major.minor` version is needed.
1. Create a new merge request and merge it.
TIP: **Tip:**
It can be handy to create the future mappings since they are pretty much known.
In that case, when a new GitLab version is released, you don't have to repeat
this first step.
### 2. Create an image for a single version
The single docs version must be created before the release merge request, but
this needs to happen when the stable branches for all products have been created.
1. Make sure you're on the root path of the `gitlab-docs` repo.
1. Run the raketask to create the single version:
```sh
./bin/rake "release:single[12.0]"
```
A new `Dockerfile.12.0` should have been created and committed to a new branch.
1. Push the newly created branch, but **don't create a merge request**.
Once you push, the `image:docker-singe` job will create a new Docker image
tagged with the branch name you created in the first step. In the end, the
image will be uploaded in the [Container Registry](https://gitlab.com/gitlab-org/gitlab-docs/container_registry)
and it will be listed under the
[`registry` environment folder](https://gitlab.com/gitlab-org/gitlab-docs/environments/folders/registry).
Optionally, you can test locally by building the image and running it:
```sh
docker build -t docs:12.0 -f Dockerfile.12.0 .
docker run -it --rm -p 4000:4000 docs:12.0
```
Visit `http://localhost:4000/12.0/` to see if everything works correctly.
### 3. Create the release merge request
Now it's time to create the monthly release merge request that adds the new
version and rotates the old one:
1. Make sure you're on the root path of the `gitlab-docs` repo.
1. Create a branch `release-X-Y`:
```sh
git checkout -b release-12-0
```
1. **Rotate the online and offline versions:**
At any given time, there are 4 browsable online versions: one pulled from
the upstream master branches (docs for GitLab.com) and the three latest
stable versions.
Edit `content/_data/versions.yaml` and rotate the versions to reflect the
new changes:
- `online`: The 3 latest stable versions.
- `offline`: All the previous versions offered as an offline archive.
1. **Add the new offline version in the 404 page redirect script:**
Since we're deprecating the oldest version each month, we need to redirect
those URLs in order not to create [404 entries](https://gitlab.com/gitlab-org/gitlab-docs/issues/221).
There's a temporary hack for now:
1. Edit `content/404.html`, making sure all offline versions under
`content/_data/versions.yaml` are in the Javascript snippet at the end of
the document.
1. **Update the `:latest` and `:archives` Docker images:**
The following two Dockerfiles need to be updated:
1. `dockerfiles/Dockerfile.archives` - Add the latest version at the top of
the list.
1. `Dockerfile.master` - Rotate the versions (oldest gets removed and latest
is added at the top of the list).
1. In the end, there should be four files in total that have changed.
Commit and push to create the merge request using the "Release" template:
```sh
git add content/ Dockerfile.master dockerfiles/Dockerfile.archives
git commit -m "Release 12.0"
git push origin release-12-0
```
### 4. Update the dropdown for all online versions
The versions dropdown is in a way "hardcoded". When the site is built, it looks
at the contents of `content/_data/versions.yaml` and based on that, the dropdown
is populated. So, older branches will have different content, which means the
dropdown will be one or more releases behind. Remember that the new changes of
the dropdown are included in the unmerged `release-X-Y` branch.
The content of `content/_data/versions.yaml` needs to change for all online
versions:
1. Before creating the merge request, [disable the scheduled pipeline](https://gitlab.com/gitlab-org/gitlab-docs/pipeline_schedules/228/edit)
by unchecking the "Active" option. Since all steps must run in sequence, we need
to do this to avoid race conditions in the event some previous versions are
updated before the release merge request is merged.
1. Run the raketask that will create all the respective merge requests needed to
update the dropdowns and will be set to automatically be merged when their
pipelines succeed. The `release-X-Y` branch needs to be present locally,
otherwise the raketask will fail:
```sh
./bin/rake release:dropdowns
```
Once all are merged, proceed to the following and final step.
TIP: **Tip:**
In case a pipeline fails, see [troubleshooting](#troubleshooting).
### 5. Merge the release merge request
The dropdown merge requests should have now been merged into their respective
version (stable branch), which will trigger another pipeline. At this point,
you need to only babysit the pipelines and make sure they don't fail:
1. Check the [pipelines page](https://gitlab.com/gitlab-org/gitlab-docs/pipelines)
and make sure all stable branches have green pipelines.
1. After all the pipelines of the online versions succeed, merge the release merge request.
1. Finally, re-activate the [scheduled pipeline](https://gitlab.com/gitlab-org/gitlab-docs/pipeline_schedules/228/edit),
save it, and hit the play button to get it started.
Once the scheduled pipeline succeeds, the docs site will be deployed with all
new versions online.
## Update an old Docker image with new upstream docs content
If there are any changes to any of the stable branches of the products that are
not included in the single Docker image, just
[rerun the pipeline](https://gitlab.com/gitlab-org/gitlab-docs/pipelines/new)
for the version in question.
## Porting new website changes to old versions
CAUTION: **Warning:**
Porting changes to older branches can have unintended effects as we're constantly
changing the backend of the website. Use only when you know what you're doing
and make sure to test locally.
The website will keep changing and being improved. In order to consolidate
those changes to the stable branches, we'd need to pick certain changes
from time to time.
If this is not possible or there are many changes, merge master into them:
```sh
git branch 12.0
git fetch origin master
git merge origin/master
```
## Troubleshooting
Releasing a new version is a long process that involves many moving parts.
### `test_internal_links_and_anchors` failing on dropdown merge requests
When [updating the dropdown for the stable versions](#4-update-the-dropdown-for-all-online-versions),
there may be cases where some links might fail. The process of how the
dropdown MRs are created have a caveat, and that is that the tests run by
pulling the master branches of all products, instead of the respective stable
ones.
In a real world scenario, the [Update 12.2 dropdown to match that of 12.4](https://gitlab.com/gitlab-org/gitlab-docs/merge_requests/604)
merge request failed because of the [`test_internal_links_and_anchors` test](https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/328042431).
This happened because there has been a rename of a product (`gitlab-monitor` to `gitlab-exporter`)
and the old name was still referenced in the 12.2 docs. If the respective stable
branches for 12.2 were used, this wouldn't have failed, but as we can see from
the [`compile_dev` job](https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/328042427),
the `master` branches were pulled.
To fix this, you need to [re-run the pipeline](https://gitlab.com/gitlab-org/gitlab-docs/pipelines/new)
for the `update-12-2-for-release-12-4` branch, by including the following environment variables:
- `BRANCH_CE` set to `12-2-stable`
- `BRANCH_EE` set to `12-2-stable-ee`
- `BRANCH_OMNIBUS` set to `12-2-stable`
- `BRANCH_RUNNER` set to `12-2-stable`
- `BRANCH_CHARTS` set to `2-2-stable`
This should make the MR pass.

8
doc/development/testing_guide/review_apps.md
View file

@ -189,10 +189,10 @@ that the `review-apps-ce/ee` cluster is unhealthy. Leading indicators may be hea
The following items may help diagnose this:
- [Instance group CPU Utilization in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-b/gke-review-apps-ee-preemp-n1-standard-8affc0f5-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_CPU&duration=P30D) - helpful to identify if nodes are problematic or the entire cluster is trending towards unhealthy
- [Instance Group size in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-b/gke-review-apps-ee-preemp-n1-standard-8affc0f5-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_SIZE&duration=P30D) - aids in identifying load spikes on the cluster. Kubernetes will add nodes up to 220 based on total resource requests.
- `kubectl top nodes --sort-by=cpu` - can identify if node spikes are common or load on specific nodes which may get rebalanced by the Kubernetes scheduler.
- `kubectl top pods --sort-by=cpu` -
- [Review Apps Health dashboard](https://app.google.stackdriver.com/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d)
- Aids in identifying load spikes on the cluster, and if nodes are problematic or the entire cluster is trending towards unhealthy.
- `kubectl top nodes | sort --key 3 --numeric` - can identify if node spikes are common or load on specific nodes which may get rebalanced by the Kubernetes scheduler.
- `kubectl top pods | sort --key 2 --numeric` -
- [K9s] - K9s is a powerful command line dashboard which allows you to filter by labels. This can help identify trends with apps exceeding the [review-app resource requests](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/review_apps/base-config.yaml). Kubernetes will schedule pods to nodes based on resource requests and allow for CPU usage up to the limits.
- In K9s you can sort or add filters by typing the `/` character
- `-lrelease=<review-app-slug>` - filters down to all pods for a release. This aids in determining what is having issues in a single deployment

6
lib/gitlab/experimentation.rb
View file

@ -43,7 +43,7 @@ module Gitlab
end
def experiment_enabled?(experiment_key)
Experimentation.enabled_for_user?(experiment_key, experimentation_subject_index)
Experimentation.enabled_for_user?(experiment_key, experimentation_subject_index) || forced_enabled?(experiment_key)
end
def track_experiment_event(experiment_key, action)
@ -94,6 +94,10 @@ module Gitlab
experiment_enabled?(experiment_key) ? 'experimental_group' : 'control_group'
end
def forced_enabled?(experiment_key)
params.has_key?(:force_experiment) && params[:force_experiment] == experiment_key.to_s
end
end
class << self

1
qa/qa/resource/user.rb
View file

@ -93,6 +93,7 @@ module QA
if Runtime::Env.signup_disabled?
self.fabricate_via_api! do |user|
user.username = username
user.password = password
end
else
self.fabricate!

32
scripts/sync-stable-branch.sh Normal file
View file

@ -0,0 +1,32 @@
#!/usr/bin/env bash
# This script triggers a merge train job to sync an EE stable branch to its
# corresponding CE stable branch.
set -e
if [[ "$MERGE_TRAIN_TRIGGER_TOKEN" == '' ]]
then
echo 'The variable MERGE_TRAIN_TRIGGER_TOKEN must be set to a non-empy value'
exit 1
fi
if [[ "$MERGE_TRAIN_TRIGGER_URL" == '' ]]
then
echo 'The variable MERGE_TRAIN_TRIGGER_URL must be set to a non-empy value'
exit 1
fi
if [[ "$CI_COMMIT_REF_NAME" == '' ]]
then
echo 'The variable CI_COMMIT_REF_NAME must be set to a non-empy value'
exit 1
fi
curl -X POST \
-F token="$MERGE_TRAIN_TRIGGER_TOKEN" \
-F ref=master \
-F "variables[MERGE_FOSS]=1" \
-F "variables[SOURCE_BRANCH]=$CI_COMMIT_REF_NAME" \
-F "variables[TARGET_BRANCH]=${CI_COMMIT_REF_NAME/-ee/}" \
"$MERGE_TRAIN_TRIGGER_URL"

18
spec/lib/gitlab/experimentation_spec.rb
View file

@ -71,6 +71,24 @@ describe Gitlab::Experimentation do
controller.experiment_enabled?(:test_experiment)
end
end
describe 'URL parameter to force enable experiment' do
context 'is not present' do
it 'returns false' do
get :index, params: { force_experiment: :test_experiment2 }
expect(controller.experiment_enabled?(:test_experiment)).to be_falsey
end
end
context 'is present' do
it 'returns true' do
get :index, params: { force_experiment: :test_experiment }
expect(controller.experiment_enabled?(:test_experiment)).to be_truthy
end
end
end
end
describe '#track_experiment_event' do