Merge branch 'docs/refactor-backup' into 'master'
Refactor the backup/restore docs See merge request gitlab-org/gitlab-ce!22576
This commit is contained in:
commit
8bc8661499
|
@ -9,37 +9,39 @@ You can only restore a backup to **exactly the same version and type (CE/EE)**
|
||||||
of GitLab on which it was created. The best way to migrate your repositories
|
of GitLab on which it was created. The best way to migrate your repositories
|
||||||
from one server to another is through backup restore.
|
from one server to another is through backup restore.
|
||||||
|
|
||||||
## Backup
|
## Requirements
|
||||||
|
|
||||||
GitLab provides a simple command line interface to backup your whole installation,
|
In order to be able to backup and restore, you need two essential tools
|
||||||
and is flexible enough to fit your needs.
|
installed on your system.
|
||||||
|
|
||||||
### Requirements
|
### Rsync
|
||||||
|
|
||||||
* rsync
|
If you installed GitLab:
|
||||||
|
|
||||||
If you're using GitLab with the Omnibus package, you're all set. If you
|
- Using the Omnibus package, you're all set.
|
||||||
installed GitLab from source, make sure you have rsync installed.
|
- From source, make sure `rsync` is installed:
|
||||||
|
|
||||||
If you're using Ubuntu, you could run:
|
```sh
|
||||||
|
# Debian/Ubuntu
|
||||||
|
sudo apt-get install rsync
|
||||||
|
|
||||||
```
|
# RHEL/CentOS
|
||||||
sudo apt-get install -y rsync
|
sudo yum install rsync
|
||||||
```
|
```
|
||||||
|
|
||||||
* tar
|
### Tar
|
||||||
|
|
||||||
Backup and restore tasks use `tar` under the hood to create and extract
|
Backup and restore tasks use `tar` under the hood to create and extract
|
||||||
archives. Ensure you have version 1.30 or above of `tar` available in your
|
archives. Ensure you have version 1.30 or above of `tar` available in your
|
||||||
system. To check the version, run:
|
system. To check the version, run:
|
||||||
|
|
||||||
```
|
```sh
|
||||||
tar --version
|
tar --version
|
||||||
```
|
```
|
||||||
|
|
||||||
### Backup timestamp
|
## Backup timestamp
|
||||||
|
|
||||||
>**Note:**
|
NOTE: **Note:**
|
||||||
In GitLab 9.2 the timestamp format was changed from `EPOCH_YYYY_MM_DD` to
|
In GitLab 9.2 the timestamp format was changed from `EPOCH_YYYY_MM_DD` to
|
||||||
`EPOCH_YYYY_MM_DD_GitLab_version`, for example `1493107454_2018_04_25`
|
`EPOCH_YYYY_MM_DD_GitLab_version`, for example `1493107454_2018_04_25`
|
||||||
would become `1493107454_2018_04_25_10.6.4-ce`.
|
would become `1493107454_2018_04_25_10.6.4-ce`.
|
||||||
|
@ -54,30 +56,46 @@ available.
|
||||||
For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`,
|
For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`,
|
||||||
then the timestamp is `1493107454_2018_04_25_10.6.4-ce`.
|
then the timestamp is `1493107454_2018_04_25_10.6.4-ce`.
|
||||||
|
|
||||||
### Creating a backup of the GitLab system
|
## Creating a backup of the GitLab system
|
||||||
|
|
||||||
|
GitLab provides a simple command line interface to backup your whole instance.
|
||||||
|
It backs up your:
|
||||||
|
|
||||||
|
- Database
|
||||||
|
- Attachments
|
||||||
|
- Git repositories data
|
||||||
|
- CI/CD job output logs
|
||||||
|
- CI/CD job artifacts
|
||||||
|
- LFS objects
|
||||||
|
- Container Registry images
|
||||||
|
- GitLab Pages content
|
||||||
|
|
||||||
|
CAUTION: **Warning:**
|
||||||
|
GitLab does not back up any configuration files, SSL certificates, or system files.
|
||||||
|
You are highly advised to [read about storing configuration files](#storing-configuration-files).
|
||||||
|
|
||||||
Use this command if you've installed GitLab with the Omnibus package:
|
Use this command if you've installed GitLab with the Omnibus package:
|
||||||
|
|
||||||
```
|
```sh
|
||||||
sudo gitlab-rake gitlab:backup:create
|
sudo gitlab-rake gitlab:backup:create
|
||||||
```
|
```
|
||||||
|
|
||||||
Use this if you've installed GitLab from source:
|
Use this if you've installed GitLab from source:
|
||||||
|
|
||||||
```
|
```sh
|
||||||
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
|
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
|
||||||
```
|
```
|
||||||
|
|
||||||
If you are running GitLab within a Docker container, you can run the backup from the host:
|
If you are running GitLab within a Docker container, you can run the backup from the host:
|
||||||
|
|
||||||
```
|
```sh
|
||||||
docker exec -t <container name> gitlab-rake gitlab:backup:create
|
docker exec -t <container name> gitlab-rake gitlab:backup:create
|
||||||
```
|
```
|
||||||
|
|
||||||
If you are using the gitlab-omnibus helm chart on a Kubernetes cluster, you can
|
If you are using the gitlab-omnibus helm chart on a Kubernetes cluster, you can
|
||||||
run the backup task on the gitlab application pod using kubectl
|
run the backup task on the gitlab application pod using kubectl:
|
||||||
|
|
||||||
```
|
```sh
|
||||||
kubectl exec -it <gitlab-gitlab pod> gitlab-rake gitlab:backup:create
|
kubectl exec -it <gitlab-gitlab pod> gitlab-rake gitlab:backup:create
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -110,9 +128,50 @@ Deleting tmp directories...[DONE]
|
||||||
Deleting old backups... [SKIPPING]
|
Deleting old backups... [SKIPPING]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Storing configuration files
|
||||||
|
|
||||||
|
A backup performed by the [raketask GitLab provides](#creating-a-backup-of-the-gitlab-system)
|
||||||
|
does **not** store your configuration files. The primary reason for this is that your
|
||||||
|
database contains encrypted information for two-factor authentication, the CI/CD
|
||||||
|
'secure variables', etc. Storing encrypted information along with its key in the
|
||||||
|
same place defeats the purpose of using encryption in the first place.
|
||||||
|
|
||||||
|
CAUTION: **Warning:**
|
||||||
|
The secrets file is essential to preserve your database encryption key.
|
||||||
|
|
||||||
|
At the very **minimum**, you must backup:
|
||||||
|
|
||||||
|
For Omnibus:
|
||||||
|
|
||||||
|
- `/etc/gitlab/gitlab-secrets.json`
|
||||||
|
- `/etc/gitlab/gitlab.rb`
|
||||||
|
|
||||||
|
For installation from source:
|
||||||
|
|
||||||
|
- `/home/git/gitlab/config/secrets.yml`
|
||||||
|
- `/home/git/gitlab/config/gitlab.yml`
|
||||||
|
|
||||||
|
For [Docker installations](https://docs.gitlab.com/omnibus/docker/), you must
|
||||||
|
back up the volume where the configuration files are stored. If you have created
|
||||||
|
the GitLab container according to the documentation, it should be under
|
||||||
|
`/srv/gitlab/config`.
|
||||||
|
|
||||||
|
You may also want to back up any TLS keys and certificates, and your
|
||||||
|
[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079).
|
||||||
|
|
||||||
|
If you use Omnibus GitLab, see some additional information
|
||||||
|
[to backup your configuration](https://docs.gitlab.com/omnibus/settings/backups.html).
|
||||||
|
|
||||||
|
In the unlikely event that the secrets file is lost, see the
|
||||||
|
[troubleshooting section](#when-the-secrets-file-is-lost).
|
||||||
|
|
||||||
|
## Backup options
|
||||||
|
|
||||||
|
The command line tool GitLab provides to backup your instance can take more options.
|
||||||
|
|
||||||
### Backup strategy option
|
### Backup strategy option
|
||||||
|
|
||||||
> **Note:** Introduced as an option in GitLab 8.17.
|
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8728) in GitLab 8.17.
|
||||||
|
|
||||||
The default backup strategy is to essentially stream data from the respective
|
The default backup strategy is to essentially stream data from the respective
|
||||||
data locations to the backup using the Linux command `tar` and `gzip`. This works
|
data locations to the backup using the Linux command `tar` and `gzip`. This works
|
||||||
|
@ -129,8 +188,11 @@ so the problem doesn't compound, but it could be a considerable change for large
|
||||||
installations. This is why the `copy` strategy is not the default in 8.17.
|
installations. This is why the `copy` strategy is not the default in 8.17.
|
||||||
|
|
||||||
To use the `copy` strategy instead of the default streaming strategy, specify
|
To use the `copy` strategy instead of the default streaming strategy, specify
|
||||||
`STRATEGY=copy` in the Rake task command. For example,
|
`STRATEGY=copy` in the Rake task command. For example:
|
||||||
`sudo gitlab-rake gitlab:backup:create STRATEGY=copy`.
|
|
||||||
|
```sh
|
||||||
|
sudo gitlab-rake gitlab:backup:create STRATEGY=copy
|
||||||
|
```
|
||||||
|
|
||||||
### Excluding specific directories from the backup
|
### Excluding specific directories from the backup
|
||||||
|
|
||||||
|
@ -151,11 +213,15 @@ Use a comma to specify several options at the same time:
|
||||||
All wikis will be backed up as part of the `repositories` group. Non-existent wikis
|
All wikis will be backed up as part of the `repositories` group. Non-existent wikis
|
||||||
will be skipped during a backup.
|
will be skipped during a backup.
|
||||||
|
|
||||||
```
|
For Omnibus GitLab packages:
|
||||||
# use this command if you've installed GitLab with the Omnibus package
|
|
||||||
sudo gitlab-rake gitlab:backup:create SKIP=db,uploads
|
|
||||||
|
|
||||||
# if you've installed GitLab from source
|
```sh
|
||||||
|
sudo gitlab-rake gitlab:backup:create SKIP=db,uploads
|
||||||
|
```
|
||||||
|
|
||||||
|
For installations from source:
|
||||||
|
|
||||||
|
```sh
|
||||||
sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production
|
sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -208,7 +274,7 @@ This example can be used for a bucket in Amsterdam (AMS3).
|
||||||
|
|
||||||
1. [Reconfigure GitLab] for the changes to take effect
|
1. [Reconfigure GitLab] for the changes to take effect
|
||||||
|
|
||||||
CAUTION: **Warning:**
|
NOTE: **Note:**
|
||||||
If you see `400 Bad Request` by using Digital Ocean Spaces, the cause may be the
|
If you see `400 Bad Request` by using Digital Ocean Spaces, the cause may be the
|
||||||
usage of backup encryption. Remove or comment the line that
|
usage of backup encryption. Remove or comment the line that
|
||||||
contains `gitlab_rails['backup_encryption']` since Digital Ocean Spaces
|
contains `gitlab_rails['backup_encryption']` since Digital Ocean Spaces
|
||||||
|
@ -370,33 +436,43 @@ backups will be copied to, and will be created if it does not exist. If the
|
||||||
directory that you want to copy the tarballs to is the root of your mounted
|
directory that you want to copy the tarballs to is the root of your mounted
|
||||||
directory, just use `.` instead.
|
directory, just use `.` instead.
|
||||||
|
|
||||||
For omnibus packages:
|
|
||||||
|
|
||||||
```ruby
|
For Omnibus GitLab packages:
|
||||||
gitlab_rails['backup_upload_connection'] = {
|
|
||||||
:provider => 'Local',
|
|
||||||
:local_root => '/mnt/backups'
|
|
||||||
}
|
|
||||||
|
|
||||||
# The directory inside the mounted folder to copy backups to
|
1. Edit `/etc/gitlab/gitlab.rb`:
|
||||||
# Use '.' to store them in the root directory
|
|
||||||
gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups'
|
```ruby
|
||||||
```
|
gitlab_rails['backup_upload_connection'] = {
|
||||||
|
:provider => 'Local',
|
||||||
|
:local_root => '/mnt/backups'
|
||||||
|
}
|
||||||
|
|
||||||
|
# The directory inside the mounted folder to copy backups to
|
||||||
|
# Use '.' to store them in the root directory
|
||||||
|
gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups'
|
||||||
|
```
|
||||||
|
|
||||||
|
1. [Reconfigure GitLab] for the changes to take effect.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
For installations from source:
|
For installations from source:
|
||||||
|
|
||||||
```yaml
|
1. Edit `home/git/gitlab/config/gitlab.yml`:
|
||||||
backup:
|
|
||||||
# snip
|
```yaml
|
||||||
upload:
|
backup:
|
||||||
# Fog storage connection settings, see http://fog.io/storage/ .
|
upload:
|
||||||
connection:
|
# Fog storage connection settings, see http://fog.io/storage/ .
|
||||||
provider: Local
|
connection:
|
||||||
local_root: '/mnt/backups'
|
provider: Local
|
||||||
# The directory inside the mounted folder to copy backups to
|
local_root: '/mnt/backups'
|
||||||
# Use '.' to store them in the root directory
|
# The directory inside the mounted folder to copy backups to
|
||||||
remote_directory: 'gitlab_backups'
|
# Use '.' to store them in the root directory
|
||||||
```
|
remote_directory: 'gitlab_backups'
|
||||||
|
```
|
||||||
|
|
||||||
|
1. [Restart GitLab] for the changes to take effect.
|
||||||
|
|
||||||
### Backup archive permissions
|
### Backup archive permissions
|
||||||
|
|
||||||
|
@ -405,45 +481,56 @@ will have owner/group git:git and 0600 permissions by default.
|
||||||
This is meant to avoid other system users reading GitLab's data.
|
This is meant to avoid other system users reading GitLab's data.
|
||||||
If you need the backup archives to have different permissions you can use the 'archive_permissions' setting.
|
If you need the backup archives to have different permissions you can use the 'archive_permissions' setting.
|
||||||
|
|
||||||
```
|
For Omnibus GitLab packages:
|
||||||
# In /etc/gitlab/gitlab.rb, for omnibus packages
|
|
||||||
gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable
|
|
||||||
```
|
|
||||||
|
|
||||||
```
|
1. Edit `/etc/gitlab/gitlab.rb`:
|
||||||
# In gitlab.yml, for installations from source:
|
|
||||||
backup:
|
|
||||||
archive_permissions: 0644 # Makes the backup archives world-readable
|
|
||||||
```
|
|
||||||
|
|
||||||
### Storing configuration files
|
```ruby
|
||||||
|
gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable
|
||||||
|
```
|
||||||
|
|
||||||
Please be informed that a backup does not store your configuration
|
1. [Reconfigure GitLab] for the changes to take effect.
|
||||||
files. One reason for this is that your database contains encrypted
|
|
||||||
information for two-factor authentication. Storing encrypted
|
|
||||||
information along with its key in the same place defeats the purpose
|
|
||||||
of using encryption in the first place!
|
|
||||||
|
|
||||||
If you use an Omnibus package please see the [instructions in the readme to backup your configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md#backup-and-restore-omnibus-gitlab-configuration).
|
---
|
||||||
If you have a cookbook installation there should be a copy of your configuration in Chef.
|
|
||||||
If you installed from source, please consider backing up your `config/secrets.yml` file, `gitlab.yml` file, any SSL keys and certificates, and your [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079).
|
|
||||||
|
|
||||||
At the very **minimum** you should backup `/etc/gitlab/gitlab.rb` and
|
For installations from source:
|
||||||
`/etc/gitlab/gitlab-secrets.json` (Omnibus), or
|
|
||||||
`/home/git/gitlab/config/secrets.yml` (source) to preserve your database
|
1. Edit `/home/git/gitlab/config/gitlab.yml`:
|
||||||
encryption key.
|
|
||||||
|
```yaml
|
||||||
|
backup:
|
||||||
|
archive_permissions: 0644 # Makes the backup archives world-readable
|
||||||
|
```
|
||||||
|
|
||||||
|
1. [Restart GitLab] for the changes to take effect.
|
||||||
|
|
||||||
### Configuring cron to make daily backups
|
### Configuring cron to make daily backups
|
||||||
|
|
||||||
>**Note:**
|
NOTE: **Note:**
|
||||||
The following cron jobs do not [backup your GitLab configuration files](#storing-configuration-files)
|
The following cron jobs do not [backup your GitLab configuration files](#storing-configuration-files)
|
||||||
or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079).
|
or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079).
|
||||||
|
|
||||||
**For Omnibus installations**
|
For Omnibus GitLab packages:
|
||||||
|
|
||||||
|
1. Edit `/etc/gitlab/gitlab.rb`:
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
## Limit backup lifetime to 7 days - 604800 seconds
|
||||||
|
gitlab_rails['backup_keep_time'] = 604800
|
||||||
|
```
|
||||||
|
|
||||||
|
1. [Reconfigure GitLab] for the changes to take effect.
|
||||||
|
|
||||||
|
Note that the `backup_keep_time` configuration option only manages local
|
||||||
|
files. GitLab does not automatically prune old files stored in a third-party
|
||||||
|
object storage (e.g., AWS S3) because the user may not have permission to list
|
||||||
|
and delete files. We recommend that you configure the appropriate retention
|
||||||
|
policy for your object storage. For example, you can configure [the S3 backup
|
||||||
|
policy as described here](http://stackoverflow.com/questions/37553070/gitlab-omnibus-delete-backup-from-amazon-s3).
|
||||||
|
|
||||||
To schedule a cron job that backs up your repositories and GitLab metadata, use the root user:
|
To schedule a cron job that backs up your repositories and GitLab metadata, use the root user:
|
||||||
|
|
||||||
```
|
```sh
|
||||||
sudo su -
|
sudo su -
|
||||||
crontab -e
|
crontab -e
|
||||||
```
|
```
|
||||||
|
@ -455,26 +542,24 @@ There, add the following line to schedule the backup for everyday at 2 AM:
|
||||||
```
|
```
|
||||||
|
|
||||||
You may also want to set a limited lifetime for backups to prevent regular
|
You may also want to set a limited lifetime for backups to prevent regular
|
||||||
backups using all your disk space. To do this add the following lines to
|
backups using all your disk space.
|
||||||
`/etc/gitlab/gitlab.rb` and reconfigure:
|
|
||||||
|
|
||||||
```
|
---
|
||||||
# limit backup lifetime to 7 days - 604800 seconds
|
|
||||||
gitlab_rails['backup_keep_time'] = 604800
|
|
||||||
```
|
|
||||||
|
|
||||||
Note that the `backup_keep_time` configuration option only manages local
|
For installations from source:
|
||||||
files. GitLab does not automatically prune old files stored in a third-party
|
|
||||||
object storage (e.g., AWS S3) because the user may not have permission to list
|
|
||||||
and delete files. We recommend that you configure the appropriate retention
|
|
||||||
policy for your object storage. For example, you can configure [the S3 backup
|
|
||||||
policy as described here](http://stackoverflow.com/questions/37553070/gitlab-omnibus-delete-backup-from-amazon-s3).
|
|
||||||
|
|
||||||
**For installation from source**
|
1. Edit `home/git/gitlab/config/gitlab.yml`:
|
||||||
|
|
||||||
```
|
```yaml
|
||||||
cd /home/git/gitlab
|
backup:
|
||||||
sudo -u git -H editor config/gitlab.yml # Enable keep_time in the backup section to automatically delete old backups
|
## Limit backup lifetime to 7 days - 604800 seconds
|
||||||
|
keep_time: 604800
|
||||||
|
```
|
||||||
|
|
||||||
|
1. [Restart GitLab] for the changes to take effect.
|
||||||
|
|
||||||
|
|
||||||
|
```sh
|
||||||
sudo -u git crontab -e # Edit the crontab for the git user
|
sudo -u git crontab -e # Edit the crontab for the git user
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -711,5 +796,53 @@ Those objects have no influence on the database backup/restore but they give thi
|
||||||
|
|
||||||
For more information see similar questions on postgresql issue tracker[here](http://www.postgresql.org/message-id/201110220712.30886.adrian.klaver@gmail.com) and [here](http://www.postgresql.org/message-id/2039.1177339749@sss.pgh.pa.us) as well as [stack overflow](http://stackoverflow.com/questions/4368789/error-must-be-owner-of-language-plpgsql).
|
For more information see similar questions on postgresql issue tracker[here](http://www.postgresql.org/message-id/201110220712.30886.adrian.klaver@gmail.com) and [here](http://www.postgresql.org/message-id/2039.1177339749@sss.pgh.pa.us) as well as [stack overflow](http://stackoverflow.com/questions/4368789/error-must-be-owner-of-language-plpgsql).
|
||||||
|
|
||||||
|
### When the secrets file is lost
|
||||||
|
|
||||||
|
If you have failed to [back up the secrets file](#storing-configuration-files),
|
||||||
|
then users with 2FA enabled will not be able to log into GitLab. In that case,
|
||||||
|
you need to [disable 2FA for everyone](../security/two_factor_authentication.md#disabling-2fa-for-everyone).
|
||||||
|
|
||||||
|
In the case of CI/CD, if your project has secure variables set, you might experience
|
||||||
|
some weird behavior, like stuck jobs or 500 errors. In that case, you can try
|
||||||
|
deleting the `ci_variables` table from the database.
|
||||||
|
|
||||||
|
CAUTION: **Warning:**
|
||||||
|
Use the following commands at your own risk, and make sure you've taken a
|
||||||
|
backup beforehand.
|
||||||
|
|
||||||
|
1. Enter the Rails console:
|
||||||
|
|
||||||
|
For Omnibus GitLab packages:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo gitlab-rails dbconsole
|
||||||
|
```
|
||||||
|
|
||||||
|
For installations from source:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Check the `ci_variables` table:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT * FROM public."ci_variables";
|
||||||
|
```
|
||||||
|
|
||||||
|
Those are the variables that you need to delete.
|
||||||
|
|
||||||
|
1. Drop the table:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
DELETE FROM ci_variables;
|
||||||
|
```
|
||||||
|
|
||||||
|
1. You may need to reconfigure or restart GitLab for the changes to take
|
||||||
|
effect.
|
||||||
|
|
||||||
|
You should now be able to visit your project, and the jobs will start
|
||||||
|
running again.
|
||||||
|
|
||||||
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
|
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
|
||||||
[restart GitLab]: ../administration/restart_gitlab.md#installations-from-source
|
[restart GitLab]: ../administration/restart_gitlab.md#installations-from-source
|
||||||
|
|
Loading…
Reference in New Issue