2019-06-07 14:33:31 +00:00
|
|
|
---
|
|
|
|
type: reference, howto
|
|
|
|
---
|
2019-07-16 07:02:20 +00:00
|
|
|
|
2019-08-02 02:41:52 +00:00
|
|
|
# Rack Attack initializer
|
|
|
|
|
|
|
|
## Overview
|
2014-05-27 12:12:15 +00:00
|
|
|
|
2019-06-07 19:27:27 +00:00
|
|
|
[Rack Attack](https://github.com/kickstarter/rack-attack), also known as Rack::Attack, is a Ruby gem
|
2018-01-23 13:07:59 +00:00
|
|
|
that is meant to protect GitLab with the ability to customize throttling and
|
2019-06-07 14:33:31 +00:00
|
|
|
to block user IP addresses.
|
|
|
|
|
2018-01-23 13:07:59 +00:00
|
|
|
You can prevent brute-force passwords attacks, scrapers, or any other offenders
|
2019-06-07 14:33:31 +00:00
|
|
|
by throttling requests from IP addresses that are making large volumes of requests.
|
|
|
|
If you find throttling is not enough to protect you against abusive clients,
|
|
|
|
Rack Attack offers IP whitelisting, blacklisting, Fail2ban style filtering, and
|
2018-01-23 13:07:59 +00:00
|
|
|
tracking.
|
2014-04-24 22:48:22 +00:00
|
|
|
|
2019-08-02 02:41:52 +00:00
|
|
|
For more information on how to use these options see the [Rack Attack README](https://github.com/kickstarter/rack-attack/blob/master/README.md).
|
|
|
|
|
|
|
|
NOTE: **Note:** See
|
|
|
|
[User and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md)
|
2019-08-07 08:44:23 +00:00
|
|
|
for simpler limits that are configured in the UI.
|
2019-08-02 02:41:52 +00:00
|
|
|
|
2019-08-07 08:44:23 +00:00
|
|
|
NOTE: **Note:** Starting with GitLab 11.2, Rack Attack is disabled by default. If your
|
2019-08-02 02:41:52 +00:00
|
|
|
instance is not exposed to the public internet, it is recommended that you leave
|
|
|
|
Rack Attack disabled.
|
|
|
|
|
|
|
|
## Behavior
|
|
|
|
|
|
|
|
If set up as described in the [Settings](#settings) section below, two behaviors
|
|
|
|
will be enabled:
|
|
|
|
|
2019-08-07 08:44:23 +00:00
|
|
|
- Protected paths will be throttled.
|
|
|
|
- Failed authentications for Git and container registry requests will trigger a temporary IP ban.
|
2019-08-02 02:41:52 +00:00
|
|
|
|
|
|
|
### Protected paths throttle
|
|
|
|
|
2019-08-07 08:44:23 +00:00
|
|
|
GitLab responds with HTTP status code `429` to POST requests at protected paths
|
|
|
|
that exceed 10 requests per minute per IP address.
|
2019-08-02 02:41:52 +00:00
|
|
|
|
|
|
|
By default, protected paths are:
|
|
|
|
|
2020-05-08 00:09:56 +00:00
|
|
|
- `/users/password`
|
|
|
|
- `/users/sign_in`
|
|
|
|
- `/api/#{API::API.version}/session.json`
|
|
|
|
- `/api/#{API::API.version}/session`
|
|
|
|
- `/users`
|
|
|
|
- `/users/confirmation`
|
|
|
|
- `/unsubscribes/`
|
|
|
|
- `/import/github/personal_access_token`
|
|
|
|
- `/admin/session`
|
2019-08-02 02:41:52 +00:00
|
|
|
|
|
|
|
This header is included in responses to blocked requests:
|
|
|
|
|
2020-03-03 03:08:31 +00:00
|
|
|
```plaintext
|
2019-08-02 02:41:52 +00:00
|
|
|
Retry-After: 60
|
|
|
|
```
|
|
|
|
|
|
|
|
For example, the following are limited to a maximum 10 requests per minute:
|
|
|
|
|
2019-08-07 08:44:23 +00:00
|
|
|
- User sign-in
|
|
|
|
- User sign-up (if enabled)
|
|
|
|
- User password reset
|
2019-08-02 02:41:52 +00:00
|
|
|
|
2019-08-07 08:44:23 +00:00
|
|
|
After 10 requests, the client must wait a minute before it can
|
|
|
|
try again.
|
2019-08-02 02:41:52 +00:00
|
|
|
|
|
|
|
### Git and container registry failed authentication ban
|
|
|
|
|
2019-08-07 08:44:23 +00:00
|
|
|
GitLab responds with HTTP status code `403` for 1 hour, if 30 failed
|
2019-08-02 02:41:52 +00:00
|
|
|
authentication requests were received in a 3-minute period from a single IP address.
|
2018-07-25 08:48:02 +00:00
|
|
|
|
2019-08-02 02:41:52 +00:00
|
|
|
This applies only to Git requests and container registry (`/jwt/auth`) requests
|
|
|
|
(combined).
|
2014-04-24 22:48:22 +00:00
|
|
|
|
2019-08-28 05:57:20 +00:00
|
|
|
This limit:
|
2013-09-26 20:40:55 +00:00
|
|
|
|
2019-08-28 05:57:20 +00:00
|
|
|
- Is reset by requests that authenticate successfully. For example, 29
|
|
|
|
failed authentication requests followed by 1 successful request, followed by 29
|
|
|
|
more failed authentication requests would not trigger a ban.
|
|
|
|
- Does not apply to JWT requests authenticated by `gitlab-ci-token`.
|
2019-08-23 19:52:53 +00:00
|
|
|
|
2019-08-02 02:41:52 +00:00
|
|
|
No response headers are provided.
|
2014-04-24 22:48:22 +00:00
|
|
|
|
2018-01-23 13:07:59 +00:00
|
|
|
## Settings
|
2014-04-24 22:48:22 +00:00
|
|
|
|
2018-01-23 13:07:59 +00:00
|
|
|
**Omnibus GitLab**
|
2014-04-24 22:48:22 +00:00
|
|
|
|
2019-06-07 14:33:31 +00:00
|
|
|
1. Open `/etc/gitlab/gitlab.rb` with your editor
|
2018-01-23 13:07:59 +00:00
|
|
|
1. Add the following:
|
2014-04-24 22:48:22 +00:00
|
|
|
|
2019-07-15 03:02:30 +00:00
|
|
|
```ruby
|
|
|
|
gitlab_rails['rack_attack_git_basic_auth'] = {
|
|
|
|
'enabled' => true,
|
|
|
|
'ip_whitelist' => ["127.0.0.1"],
|
|
|
|
'maxretry' => 10, # Limit the number of Git HTTP authentication attempts per IP
|
|
|
|
'findtime' => 60, # Reset the auth attempt counter per IP after 60 seconds
|
|
|
|
'bantime' => 3600 # Ban an IP for one hour (3600s) after too many auth attempts
|
|
|
|
}
|
|
|
|
```
|
2013-09-26 20:40:55 +00:00
|
|
|
|
2018-11-13 00:39:21 +00:00
|
|
|
1. Reconfigure GitLab:
|
2013-09-26 20:40:55 +00:00
|
|
|
|
2020-03-03 03:08:31 +00:00
|
|
|
```shell
|
2019-07-15 03:02:30 +00:00
|
|
|
sudo gitlab-ctl reconfigure
|
|
|
|
```
|
2013-09-26 20:40:55 +00:00
|
|
|
|
2018-01-23 13:07:59 +00:00
|
|
|
The following settings can be configured:
|
2013-09-26 20:40:55 +00:00
|
|
|
|
2018-01-25 12:34:57 +00:00
|
|
|
- `enabled`: By default this is set to `false`. Set this to `true` to enable Rack Attack.
|
2019-06-27 22:44:46 +00:00
|
|
|
- `ip_whitelist`: Whitelist any IPs from being blocked. They must be formatted as strings within a Ruby array.
|
2019-07-15 03:02:30 +00:00
|
|
|
CIDR notation is supported in GitLab v12.1 and up.
|
|
|
|
For example, `["127.0.0.1", "127.0.0.2", "127.0.0.3", "192.168.0.1/24"]`.
|
2018-01-23 13:07:59 +00:00
|
|
|
- `maxretry`: The maximum amount of times a request can be made in the
|
2019-07-15 03:02:30 +00:00
|
|
|
specified time.
|
2019-06-07 14:33:31 +00:00
|
|
|
- `findtime`: The maximum amount of time that failed requests can count against an IP
|
2019-07-15 03:02:30 +00:00
|
|
|
before it's blacklisted (in seconds).
|
2018-11-20 16:44:15 +00:00
|
|
|
- `bantime`: The total amount of time that a blacklisted IP will be blocked (in
|
2019-07-15 03:02:30 +00:00
|
|
|
seconds).
|
2018-01-23 13:07:59 +00:00
|
|
|
|
|
|
|
**Installations from source**
|
|
|
|
|
|
|
|
These settings can be found in `config/initializers/rack_attack.rb`. If you are
|
|
|
|
missing `config/initializers/rack_attack.rb`, the following steps need to be
|
|
|
|
taken in order to enable protection for your GitLab instance:
|
|
|
|
|
|
|
|
1. In `config/application.rb` find and uncomment the following line:
|
|
|
|
|
2019-07-15 03:02:30 +00:00
|
|
|
```ruby
|
|
|
|
config.middleware.use Rack::Attack
|
|
|
|
```
|
2018-01-23 13:07:59 +00:00
|
|
|
|
|
|
|
1. Restart GitLab:
|
|
|
|
|
2020-01-30 15:09:15 +00:00
|
|
|
```shell
|
2019-07-15 03:02:30 +00:00
|
|
|
sudo service gitlab restart
|
|
|
|
```
|
2018-01-23 13:07:59 +00:00
|
|
|
|
|
|
|
If you want more restrictive/relaxed throttle rules, edit
|
|
|
|
`config/initializers/rack_attack.rb` and change the `limit` or `period` values.
|
|
|
|
For example, more relaxed throttle rules will be if you set
|
|
|
|
`limit: 3` and `period: 1.seconds` (this will allow 3 requests per second).
|
|
|
|
You can also add other paths to the protected list by adding to `paths_to_be_protected`
|
2019-08-07 08:44:23 +00:00
|
|
|
variable. If you change any of these settings you must restart your
|
2018-01-23 13:07:59 +00:00
|
|
|
GitLab instance.
|
|
|
|
|
|
|
|
## Remove blocked IPs from Rack Attack via Redis
|
|
|
|
|
|
|
|
In case you want to remove a blocked IP, follow these steps:
|
|
|
|
|
|
|
|
1. Find the IPs that have been blocked in the production log:
|
|
|
|
|
2020-01-30 15:09:15 +00:00
|
|
|
```shell
|
2019-07-15 03:02:30 +00:00
|
|
|
grep "Rack_Attack" /var/log/gitlab/gitlab-rails/auth.log
|
|
|
|
```
|
2018-01-23 13:07:59 +00:00
|
|
|
|
2018-11-13 00:39:21 +00:00
|
|
|
1. Since the blacklist is stored in Redis, you need to open up `redis-cli`:
|
2018-01-23 13:07:59 +00:00
|
|
|
|
2020-01-30 15:09:15 +00:00
|
|
|
```shell
|
2019-07-15 03:02:30 +00:00
|
|
|
/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket
|
|
|
|
```
|
2018-01-23 13:07:59 +00:00
|
|
|
|
2018-11-13 00:39:21 +00:00
|
|
|
1. You can remove the block using the following syntax, replacing `<ip>` with
|
2018-01-23 13:07:59 +00:00
|
|
|
the actual IP that is blacklisted:
|
|
|
|
|
2020-03-25 06:07:58 +00:00
|
|
|
```plaintext
|
2019-07-15 03:02:30 +00:00
|
|
|
del cache:gitlab:rack::attack:allow2ban:ban:<ip>
|
|
|
|
```
|
2018-01-23 13:07:59 +00:00
|
|
|
|
2018-11-13 00:39:21 +00:00
|
|
|
1. Confirm that the key with the IP no longer shows up:
|
2018-01-23 13:07:59 +00:00
|
|
|
|
2020-03-25 06:07:58 +00:00
|
|
|
```plaintext
|
2019-07-15 03:02:30 +00:00
|
|
|
keys *rack::attack*
|
|
|
|
```
|
2018-01-23 13:07:59 +00:00
|
|
|
|
2018-11-13 00:39:21 +00:00
|
|
|
1. Optionally, add the IP to the whitelist to prevent it from being blacklisted
|
2018-01-23 13:07:59 +00:00
|
|
|
again (see [settings](#settings)).
|
|
|
|
|
|
|
|
## Troubleshooting
|
|
|
|
|
|
|
|
### Rack attack is blacklisting the load balancer
|
|
|
|
|
|
|
|
Rack Attack may block your load balancer if all traffic appears to come from
|
|
|
|
the load balancer. In that case, you will need to:
|
|
|
|
|
|
|
|
1. [Configure `nginx[real_ip_trusted_addresses]`](https://docs.gitlab.com/omnibus/settings/nginx.html#configuring-gitlab-trusted_proxies-and-the-nginx-real_ip-module).
|
|
|
|
This will keep users' IPs from being listed as the load balancer IPs.
|
2018-11-13 00:39:21 +00:00
|
|
|
1. Whitelist the load balancer's IP address(es) in the Rack Attack [settings](#settings).
|
|
|
|
1. Reconfigure GitLab:
|
2018-01-23 13:07:59 +00:00
|
|
|
|
2020-03-03 03:08:31 +00:00
|
|
|
```shell
|
2019-07-15 03:02:30 +00:00
|
|
|
sudo gitlab-ctl reconfigure
|
|
|
|
```
|
2018-01-23 13:07:59 +00:00
|
|
|
|
2018-11-13 00:39:21 +00:00
|
|
|
1. [Remove the block via Redis.](#remove-blocked-ips-from-rack-attack-via-redis)
|