Remove MySQL references from development docs
I noticed the doc/development/testing_guide/best_practices.md still referenced the `[run mysql]` tags, etc. They no longer work, so I removed them, then realised I had better clean up the rest of doc/development !
This commit is contained in:
parent
95075fee78
commit
5a574883f9
10 changed files with 34 additions and 72 deletions
|
@ -20,7 +20,7 @@ A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a w
|
||||||
|
|
||||||
We also support deploying GitLab on Kubernetes using our [gitlab Helm chart](https://docs.gitlab.com/charts/).
|
We also support deploying GitLab on Kubernetes using our [gitlab Helm chart](https://docs.gitlab.com/charts/).
|
||||||
|
|
||||||
The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository.
|
The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository.
|
||||||
|
|
||||||
When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects.
|
When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects.
|
||||||
|
|
||||||
|
@ -511,7 +511,15 @@ To summarize here's the [directory structure of the `git` user home directory](.
|
||||||
ps aux | grep '^git'
|
ps aux | grep '^git'
|
||||||
```
|
```
|
||||||
|
|
||||||
GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or Nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process).
|
GitLab has several components to operate. It requires a persistent database
|
||||||
|
(PostgreSQL) and redis database, and uses Apache httpd or Nginx to proxypass
|
||||||
|
Unicorn. All these components should run as different system users to GitLab
|
||||||
|
(e.g., `postgres`, `redis` and `www-data`, instead of `git`).
|
||||||
|
|
||||||
|
As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server
|
||||||
|
running on port `8080` by default). Under the GitLab user there are normally 4
|
||||||
|
processes: `unicorn_rails master` (1 process), `unicorn_rails worker`
|
||||||
|
(2 processes), `sidekiq` (1 process).
|
||||||
|
|
||||||
### Repository access
|
### Repository access
|
||||||
|
|
||||||
|
@ -554,12 +562,9 @@ $ /etc/init.d/nginx
|
||||||
Usage: nginx {start|stop|restart|reload|force-reload|status|configtest}
|
Usage: nginx {start|stop|restart|reload|force-reload|status|configtest}
|
||||||
```
|
```
|
||||||
|
|
||||||
Persistent database (one of the following)
|
Persistent database
|
||||||
|
|
||||||
```
|
```
|
||||||
/etc/init.d/mysqld
|
|
||||||
Usage: /etc/init.d/mysqld {start|stop|status|restart|condrestart|try-restart|reload|force-reload}
|
|
||||||
|
|
||||||
$ /etc/init.d/postgresql
|
$ /etc/init.d/postgresql
|
||||||
Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..]
|
Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..]
|
||||||
```
|
```
|
||||||
|
@ -597,11 +602,6 @@ PostgreSQL
|
||||||
|
|
||||||
- `/var/log/postgresql/*`
|
- `/var/log/postgresql/*`
|
||||||
|
|
||||||
MySQL
|
|
||||||
|
|
||||||
- `/var/log/mysql/*`
|
|
||||||
- `/var/log/mysql.*`
|
|
||||||
|
|
||||||
### GitLab specific config files
|
### GitLab specific config files
|
||||||
|
|
||||||
GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include:
|
GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include:
|
||||||
|
|
|
@ -1,6 +1,6 @@
|
||||||
# Hash Indexes
|
# Hash Indexes
|
||||||
|
|
||||||
Both PostgreSQL and MySQL support hash indexes besides the regular btree
|
PostgreSQL supports hash indexes besides the regular btree
|
||||||
indexes. Hash indexes however are to be avoided at all costs. While they may
|
indexes. Hash indexes however are to be avoided at all costs. While they may
|
||||||
_sometimes_ provide better performance the cost of rehashing can be very high.
|
_sometimes_ provide better performance the cost of rehashing can be very high.
|
||||||
More importantly: at least until PostgreSQL 10.0 hash indexes are not
|
More importantly: at least until PostgreSQL 10.0 hash indexes are not
|
||||||
|
|
|
@ -9,7 +9,7 @@ bundle exec rake setup
|
||||||
```
|
```
|
||||||
|
|
||||||
The `setup` task is an alias for `gitlab:setup`.
|
The `setup` task is an alias for `gitlab:setup`.
|
||||||
This tasks calls `db:reset` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and finally it calls `db:seed_fu` to seed the database.
|
This tasks calls `db:reset` to create the database, and calls `db:seed_fu` to seed the database.
|
||||||
Note: `db:setup` calls `db:seed` but this does nothing.
|
Note: `db:setup` calls `db:seed` but this does nothing.
|
||||||
|
|
||||||
### Seeding issues for all or a given project
|
### Seeding issues for all or a given project
|
||||||
|
|
|
@ -2,7 +2,7 @@
|
||||||
|
|
||||||
Storing SHA1 hashes as strings is not very space efficient. A SHA1 as a string
|
Storing SHA1 hashes as strings is not very space efficient. A SHA1 as a string
|
||||||
requires at least 40 bytes, an additional byte to store the encoding, and
|
requires at least 40 bytes, an additional byte to store the encoding, and
|
||||||
perhaps more space depending on the internals of PostgreSQL and MySQL.
|
perhaps more space depending on the internals of PostgreSQL.
|
||||||
|
|
||||||
On the other hand, if one were to store a SHA1 as binary one would only need 20
|
On the other hand, if one were to store a SHA1 as binary one would only need 20
|
||||||
bytes for the actual SHA1, and 1 or 4 bytes of additional space (again depending
|
bytes for the actual SHA1, and 1 or 4 bytes of additional space (again depending
|
||||||
|
|
|
@ -15,14 +15,11 @@ FROM issues
|
||||||
WHERE title LIKE 'WIP:%';
|
WHERE title LIKE 'WIP:%';
|
||||||
```
|
```
|
||||||
|
|
||||||
On PostgreSQL the `LIKE` statement is case-sensitive. On MySQL this depends on
|
On PostgreSQL the `LIKE` statement is case-sensitive. To perform a case-insensitive
|
||||||
the case-sensitivity of the collation, which is usually case-insensitive. To
|
`LIKE` you have to use `ILIKE` instead.
|
||||||
perform a case-insensitive `LIKE` on PostgreSQL you have to use `ILIKE` instead.
|
|
||||||
This statement in turn isn't supported on MySQL.
|
|
||||||
|
|
||||||
To work around this problem you should write `LIKE` queries using Arel instead
|
To handle this automatically you should use `LIKE` queries using Arel instead
|
||||||
of raw SQL fragments as Arel automatically uses `ILIKE` on PostgreSQL and `LIKE`
|
of raw SQL fragments, as Arel automatically uses `ILIKE` on PostgreSQL.
|
||||||
on MySQL. This means that instead of this:
|
|
||||||
|
|
||||||
```ruby
|
```ruby
|
||||||
Issue.where('title LIKE ?', 'WIP:%')
|
Issue.where('title LIKE ?', 'WIP:%')
|
||||||
|
@ -45,7 +42,7 @@ table = Issue.arel_table
|
||||||
Issue.where(table[:title].matches('WIP:%').or(table[:foo].matches('WIP:%')))
|
Issue.where(table[:title].matches('WIP:%').or(table[:foo].matches('WIP:%')))
|
||||||
```
|
```
|
||||||
|
|
||||||
For PostgreSQL this produces:
|
On PostgreSQL, this produces:
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
SELECT *
|
SELECT *
|
||||||
|
@ -53,18 +50,10 @@ FROM issues
|
||||||
WHERE (title ILIKE 'WIP:%' OR foo ILIKE 'WIP:%')
|
WHERE (title ILIKE 'WIP:%' OR foo ILIKE 'WIP:%')
|
||||||
```
|
```
|
||||||
|
|
||||||
In turn for MySQL this produces:
|
|
||||||
|
|
||||||
```sql
|
|
||||||
SELECT *
|
|
||||||
FROM issues
|
|
||||||
WHERE (title LIKE 'WIP:%' OR foo LIKE 'WIP:%')
|
|
||||||
```
|
|
||||||
|
|
||||||
## LIKE & Indexes
|
## LIKE & Indexes
|
||||||
|
|
||||||
Neither PostgreSQL nor MySQL use any indexes when using `LIKE` / `ILIKE` with a
|
PostgreSQL won't use any indexes when using `LIKE` / `ILIKE` with a wildcard at
|
||||||
wildcard at the start. For example, this will not use any indexes:
|
the start. For example, this will not use any indexes:
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
SELECT *
|
SELECT *
|
||||||
|
@ -75,9 +64,8 @@ WHERE title ILIKE '%WIP:%';
|
||||||
Because the value for `ILIKE` starts with a wildcard the database is not able to
|
Because the value for `ILIKE` starts with a wildcard the database is not able to
|
||||||
use an index as it doesn't know where to start scanning the indexes.
|
use an index as it doesn't know where to start scanning the indexes.
|
||||||
|
|
||||||
MySQL provides no known solution to this problem. Luckily PostgreSQL _does_
|
Luckily, PostgreSQL _does_ provide a solution: trigram GIN indexes. These
|
||||||
provide a solution: trigram GIN indexes. These indexes can be created as
|
indexes can be created as follows:
|
||||||
follows:
|
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
CREATE INDEX [CONCURRENTLY] index_name_here
|
CREATE INDEX [CONCURRENTLY] index_name_here
|
||||||
|
|
|
@ -15,16 +15,6 @@ manifest themselves within our code. When designing our tests, take time to revi
|
||||||
our test design. We can find some helpful heuristics documented in the Handbook in the
|
our test design. We can find some helpful heuristics documented in the Handbook in the
|
||||||
[Test Design](https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/test-design/) section.
|
[Test Design](https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/test-design/) section.
|
||||||
|
|
||||||
## Run tests against MySQL
|
|
||||||
|
|
||||||
By default, tests are only run against PostgreSQL, but you can run them on
|
|
||||||
demand against MySQL by following one of the following conventions:
|
|
||||||
|
|
||||||
| Convention | Valid example |
|
|
||||||
|:----------------------|:-----------------------------|
|
|
||||||
| Include `mysql` in your branch name | `enhance-mysql-support` |
|
|
||||||
| Include `[run mysql]` in your commit message | `Fix MySQL support<br><br>[run mysql]` |
|
|
||||||
|
|
||||||
## Test speed
|
## Test speed
|
||||||
|
|
||||||
GitLab has a massive test suite that, without [parallelization], can take hours
|
GitLab has a massive test suite that, without [parallelization], can take hours
|
||||||
|
|
|
@ -39,7 +39,6 @@ slowest test files and try to improve them.
|
||||||
|
|
||||||
## CI setup
|
## CI setup
|
||||||
|
|
||||||
- On CE and EE, the test suite runs both PostgreSQL and MySQL.
|
|
||||||
- Rails logging to `log/test.log` is disabled by default in CI [for
|
- Rails logging to `log/test.log` is disabled by default in CI [for
|
||||||
performance reasons][logging]. To override this setting, provide the
|
performance reasons][logging]. To override this setting, provide the
|
||||||
`RAILS_ENABLE_TEST_LOG` environment variable.
|
`RAILS_ENABLE_TEST_LOG` environment variable.
|
||||||
|
|
|
@ -35,8 +35,8 @@ Once a test is in quarantine, there are 3 choices:
|
||||||
|
|
||||||
Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:
|
Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:
|
||||||
|
|
||||||
- `rspec-pg-quarantine` and `rspec-mysql-quarantine` (CE & EE)
|
- `rspec-pg-quarantine` (CE & EE)
|
||||||
- `rspec-pg-quarantine-ee` and `rspec-mysql-quarantine-ee` (EE only)
|
- `rspec-pg-quarantine-ee` (EE only)
|
||||||
|
|
||||||
## Automatic retries and flaky tests detection
|
## Automatic retries and flaky tests detection
|
||||||
|
|
||||||
|
|
|
@ -1,15 +1,15 @@
|
||||||
# Verifying Database Capabilities
|
# Verifying Database Capabilities
|
||||||
|
|
||||||
Sometimes certain bits of code may only work on a certain database and/or
|
Sometimes certain bits of code may only work on a certain database
|
||||||
version. While we try to avoid such code as much as possible sometimes it is
|
version. While we try to avoid such code as much as possible sometimes it is
|
||||||
necessary to add database (version) specific behaviour.
|
necessary to add database (version) specific behaviour.
|
||||||
|
|
||||||
To facilitate this we have the following methods that you can use:
|
To facilitate this we have the following methods that you can use:
|
||||||
|
|
||||||
- `Gitlab::Database.postgresql?`: returns `true` if PostgreSQL is being used
|
- `Gitlab::Database.postgresql?`: returns `true` if PostgreSQL is being used.
|
||||||
- `Gitlab::Database.mysql?`: returns `true` if MySQL is being used
|
You can normally just assume this is the case.
|
||||||
- `Gitlab::Database.version`: returns the PostgreSQL version number as a string
|
- `Gitlab::Database.version`: returns the PostgreSQL version number as a string
|
||||||
in the format `X.Y.Z`. This method does not work for MySQL
|
in the format `X.Y.Z`.
|
||||||
|
|
||||||
This allows you to write code such as:
|
This allows you to write code such as:
|
||||||
|
|
||||||
|
|
|
@ -7,9 +7,8 @@ downtime.
|
||||||
|
|
||||||
## Adding Columns
|
## Adding Columns
|
||||||
|
|
||||||
On PostgreSQL you can safely add a new column to an existing table as long as it
|
You can safely add a new column to an existing table as long as it does **not**
|
||||||
does **not** have a default value. For example, this query would not require
|
have a default value. For example, this query would not require downtime:
|
||||||
downtime:
|
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
ALTER TABLE projects ADD COLUMN random_value int;
|
ALTER TABLE projects ADD COLUMN random_value int;
|
||||||
|
@ -27,11 +26,6 @@ This requires updating every single row in the `projects` table so that
|
||||||
indexes in a table. This in turn acquires enough locks on the table for it to
|
indexes in a table. This in turn acquires enough locks on the table for it to
|
||||||
effectively block any other queries.
|
effectively block any other queries.
|
||||||
|
|
||||||
As of MySQL 5.6 adding a column to a table is still quite an expensive
|
|
||||||
operation, even when using `ALGORITHM=INPLACE` and `LOCK=NONE`. This means
|
|
||||||
downtime _may_ be required when modifying large tables as otherwise the
|
|
||||||
operation could potentially take hours to complete.
|
|
||||||
|
|
||||||
Adding a column with a default value _can_ be done without requiring downtime
|
Adding a column with a default value _can_ be done without requiring downtime
|
||||||
when using the migration helper method
|
when using the migration helper method
|
||||||
`Gitlab::Database::MigrationHelpers#add_column_with_default`. This method works
|
`Gitlab::Database::MigrationHelpers#add_column_with_default`. This method works
|
||||||
|
@ -311,8 +305,7 @@ migrations](background_migrations.md#cleaning-up).
|
||||||
## Adding Indexes
|
## Adding Indexes
|
||||||
|
|
||||||
Adding indexes is an expensive process that blocks INSERT and UPDATE queries for
|
Adding indexes is an expensive process that blocks INSERT and UPDATE queries for
|
||||||
the duration. When using PostgreSQL one can work around this by using the
|
the duration. You can work around this by using the `CONCURRENTLY` option:
|
||||||
`CONCURRENTLY` option:
|
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
CREATE INDEX CONCURRENTLY index_name ON projects (column_name);
|
CREATE INDEX CONCURRENTLY index_name ON projects (column_name);
|
||||||
|
@ -336,17 +329,9 @@ end
|
||||||
Note that `add_concurrent_index` can not be reversed automatically, thus you
|
Note that `add_concurrent_index` can not be reversed automatically, thus you
|
||||||
need to manually define `up` and `down`.
|
need to manually define `up` and `down`.
|
||||||
|
|
||||||
When running this on PostgreSQL the `CONCURRENTLY` option mentioned above is
|
|
||||||
used. On MySQL this method produces a regular `CREATE INDEX` query.
|
|
||||||
|
|
||||||
MySQL doesn't really have a workaround for this. Supposedly it _can_ create
|
|
||||||
indexes without the need for downtime but only for variable width columns. The
|
|
||||||
details on this are a bit sketchy. Since it's better to be safe than sorry one
|
|
||||||
should assume that adding indexes requires downtime on MySQL.
|
|
||||||
|
|
||||||
## Dropping Indexes
|
## Dropping Indexes
|
||||||
|
|
||||||
Dropping an index does not require downtime on both PostgreSQL and MySQL.
|
Dropping an index does not require downtime.
|
||||||
|
|
||||||
## Adding Tables
|
## Adding Tables
|
||||||
|
|
||||||
|
@ -370,7 +355,7 @@ transaction this means this approach would require downtime.
|
||||||
|
|
||||||
GitLab allows you to work around this by using
|
GitLab allows you to work around this by using
|
||||||
`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method
|
`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method
|
||||||
ensures that when PostgreSQL is used no downtime is needed.
|
ensures that no downtime is needed.
|
||||||
|
|
||||||
## Removing Foreign Keys
|
## Removing Foreign Keys
|
||||||
|
|
||||||
|
|
Loading…
Reference in a new issue