Merge branch 'docs-debug-harishsr' into 'master'
Port 3 more debug docs to the docs project Closes #63035 See merge request gitlab-org/gitlab-ce!31368
This commit is contained in:
commit
5512dc23de
|
@ -187,13 +187,29 @@ Learn how to install, configure, update, and maintain your GitLab instance.
|
|||
- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong
|
||||
- [Log system](logs.md): Where to look for logs.
|
||||
- [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs.
|
||||
- Useful [diagnostics tools](troubleshooting/diagnostics_tools.md) that are sometimes used by the GitLab
|
||||
Support team.
|
||||
- [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md): Tips to troubleshoot ElasticSearch.
|
||||
- [Kubernetes troubleshooting](troubleshooting/kubernetes_cheat_sheet.md): Commands and tips useful
|
||||
for troubleshooting Kubernetes-related issues.
|
||||
- Useful links from the Support Team:
|
||||
- [GitLab Developer Docs](https://docs.gitlab.com/ee/development/README.html).
|
||||
- [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html).
|
||||
- [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html).
|
||||
- [Strace zine](https://wizardzines.com/zines/strace/).
|
||||
- [Troubleshooting ElasticSearch](troubleshooting/elasticsearch.md)
|
||||
|
||||
### Support Team Docs
|
||||
|
||||
The GitLab Support Team has collected a lot of information about troubleshooting GitLab
|
||||
instances. These documents are normally used by the Support Team itself, or by customers
|
||||
with direct guidance from a Support Team member. GitLab administrators may find the
|
||||
information useful for troubleshooting, but if you are experiencing trouble with your
|
||||
GitLab instance, you should check your [support options](https://about.gitlab.com/support/)
|
||||
before referring to these documents.
|
||||
|
||||
CAUTION: **Warning:**
|
||||
Using the commands listed in the documentation below could result in data loss or
|
||||
other damage to a GitLab instance, and should only be used by experienced administrators
|
||||
who are aware of the risks.
|
||||
|
||||
- [Useful diagnostics tools](troubleshooting/diagnostics_tools.md)
|
||||
- [Useful Linux commands](troubleshooting/linux_cheat_sheet.md)
|
||||
- [Troubleshooting Kubernetes](troubleshooting/kubernetes_cheat_sheet.md)
|
||||
- [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers)
|
||||
- [GitLab rails console commands](troubleshooting/gitlab_rails_cheat_sheet.md) (for Support Engineers)
|
||||
- Useful links:
|
||||
- [GitLab Developer Docs](../development/README.md)
|
||||
- [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html)
|
||||
- [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html)
|
||||
- [Strace zine](https://wizardzines.com/zines/strace/)
|
||||
|
|
File diff suppressed because it is too large
Load Diff
|
@ -0,0 +1,339 @@
|
|||
---
|
||||
type: reference
|
||||
---
|
||||
|
||||
# Linux Cheat Sheet
|
||||
|
||||
This is the GitLab Support Team's collection of information regarding Linux, that they
|
||||
sometimes use while troubleshooting. It is listed here for transparency,
|
||||
and it may be useful for users with experience with Linux. If you are currently
|
||||
having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/)
|
||||
first, before attempting to use this information.
|
||||
|
||||
CAUTION: **CAUTION:**
|
||||
If you are administering GitLab you are expected to know these commands for your distribution
|
||||
of choice. If you are a GitLab Support Engineer, consider this a cross-reference to
|
||||
translate `yum` -> `apt-get` and the like.
|
||||
|
||||
Note: **Note:**
|
||||
Most of the commands below have not been labeled as to which distribution they work
|
||||
on. Contributions are welcome to help add them.
|
||||
|
||||
## System Commands
|
||||
|
||||
### Distro Information
|
||||
|
||||
```bash
|
||||
# Debian/Ubuntu
|
||||
uname -a
|
||||
lsb_release -a
|
||||
|
||||
# CentOS/RedHat
|
||||
cat /etc/centos-release
|
||||
cat /etc/redhat-release
|
||||
|
||||
# This will provide a lot more information
|
||||
cat /etc/os-release
|
||||
```
|
||||
|
||||
### Shut down or Reboot
|
||||
|
||||
```bash
|
||||
shutdown -h now
|
||||
reboot
|
||||
```
|
||||
|
||||
### Permissions
|
||||
|
||||
```bash
|
||||
# change the user:group ownership of a file/dir
|
||||
chown root:git <file_or_dir>
|
||||
|
||||
# make a file executable
|
||||
chmod u+x <file>
|
||||
```
|
||||
|
||||
### Files & Dirs
|
||||
|
||||
```bash
|
||||
# create a new directory and all subdirectories
|
||||
mkdir -p dir/dir2/dir3
|
||||
|
||||
# Send a command's output to file.txt, no STDOUT
|
||||
ls > file.txt
|
||||
|
||||
# Send a command's output to file.txt AND see it in STDOUT
|
||||
ls | tee /tmp/file.txt
|
||||
|
||||
# Search and Replace within a file
|
||||
sed -i 's/original-text/new-text/g' <filename>
|
||||
```
|
||||
|
||||
### See all set environment variables
|
||||
|
||||
```bash
|
||||
env
|
||||
```
|
||||
|
||||
## Searching
|
||||
|
||||
### File names
|
||||
|
||||
```bash
|
||||
# search for a file in a filesystem
|
||||
find . -name 'filename.rb' -print
|
||||
|
||||
# locate a file
|
||||
locate <filename>
|
||||
|
||||
# see command history
|
||||
history
|
||||
|
||||
# search CLI history
|
||||
<ctrl>-R
|
||||
```
|
||||
|
||||
### File contents
|
||||
|
||||
```bash
|
||||
# -B/A = show 2 lines before/after search_term
|
||||
grep -B 2 -A 2 search_term <filename>
|
||||
|
||||
# -<number> shows both before and after
|
||||
grep -2 search_term <filename>
|
||||
|
||||
# Search on all files in directory (recursively)
|
||||
grep -r search_term <directory>
|
||||
|
||||
# search through *.gz files is the same except with zgrep
|
||||
zgrep search_term <filename>
|
||||
|
||||
# Fast grep printing lines containing a string pattern
|
||||
fgrep -R string_pattern <filename or directory>
|
||||
```
|
||||
|
||||
### CLI
|
||||
|
||||
```bash
|
||||
# View command history
|
||||
history
|
||||
|
||||
# Run last command that started with 'his' (3 letters min)
|
||||
!his
|
||||
|
||||
# Search through command history
|
||||
<ctrl>-R
|
||||
|
||||
# Execute last command with sudo
|
||||
sudo !!
|
||||
```
|
||||
|
||||
## Managing resources
|
||||
|
||||
### Memory, Disk, & CPU usage
|
||||
|
||||
```bash
|
||||
# disk space info. The '-h' gives the data in human-readable values
|
||||
df -h
|
||||
|
||||
# size of each file/dir and its contents in the current dir
|
||||
du -hd 1
|
||||
|
||||
# or alternative
|
||||
du -h --max-depth=1
|
||||
|
||||
# find files greater than certain size(k, M, G) and list them in order
|
||||
# get rid of the + for exact, - for less than
|
||||
find / -type f -size +100M -print0 | xargs -0 du -hs | sort -h
|
||||
|
||||
# Find free memory on a system
|
||||
free -m
|
||||
|
||||
# Find what processes are using memory/CPU and organize by it
|
||||
# Load average is 1/CPU for 1, 5, and 15 minutes
|
||||
top -o %MEM
|
||||
top -o %CPU
|
||||
```
|
||||
|
||||
### Strace
|
||||
|
||||
```bash
|
||||
# strace a process
|
||||
strace -tt -T -f -y -s 1024 -p <pid>
|
||||
|
||||
# -tt print timestamps with microsecond accuracy
|
||||
|
||||
# -T print the time spent in each syscall
|
||||
|
||||
# -f also trace any child processes that forked
|
||||
|
||||
# -y print the path associated with file handles
|
||||
|
||||
# -s max string length to print for an event
|
||||
|
||||
# -o output file
|
||||
|
||||
# run strace on all unicorn processes
|
||||
ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -s 1024 -o /tmp/unicorn.txt
|
||||
```
|
||||
|
||||
See the [strace zine](https://wizardzines.com/zines/strace/) for a quick walkthrough.
|
||||
|
||||
Brendan Gregg has a more detailed explanation of [how to use strace](http://www.brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html).
|
||||
|
||||
Be aware that strace can have major impacts to system performance when it is running.
|
||||
|
||||
### The Strace Parser tool
|
||||
|
||||
Our [strace-parser tool](https://gitlab.com/wchandler/strace-parser) can be used to
|
||||
provide a high level summary of the `strace` output. It is similar to `strace -C`,
|
||||
but provides much more detailed statistics.
|
||||
|
||||
MacOS and Linux binaries [are available](https://gitlab.com/gitlab-com/support/toolbox/strace-parser/-/tags),
|
||||
or you can build it from source if you have the Rust compiler.
|
||||
|
||||
#### How to use the tool
|
||||
|
||||
First run the tool with no arguments other than the strace output file name to get
|
||||
a summary of the top processes sorted by time spent actively performing tasks. You
|
||||
can also sort based on total time, # of syscalls made, PID #, and # of child processes
|
||||
using the `-S` or `--sort` flag. The number of results defaults to 25 processes, but
|
||||
can be changed using the `-c`/`--count` option. See `--help` for full details.
|
||||
|
||||
```sh
|
||||
$ ./strace-parser strace.txt
|
||||
|
||||
Top 25 PIDs
|
||||
-----------
|
||||
|
||||
pid active (ms) wait (ms) total (ms) % active syscalls
|
||||
---------- ---------- --------- --------- --------- ---------
|
||||
8795 689.072 45773.832 46462.902 16.89% 23018
|
||||
13408 679.432 55910.891 56590.320 16.65% 28593
|
||||
6423 554.822 13175.485 13730.308 13.60% 13735
|
||||
...
|
||||
```
|
||||
|
||||
Based on the summary, you can then view the details of syscalls made by one or more
|
||||
procsses using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags for
|
||||
a sorted list. `--stats` takes the same sorting and count options as summary.
|
||||
|
||||
```sh
|
||||
$ ./strace-parse strace.text -p 6423
|
||||
|
||||
PID 6423
|
||||
13735 syscalls, active time: 554.822ms, total time: 13730.308ms
|
||||
|
||||
syscall count total max avg min errors
|
||||
(ms) (ms) (ms) (ms)
|
||||
--------------- -------- ---------- ---------- ---------- ---------- --------
|
||||
epoll_wait 628 13175.485 21.259 20.980 0.020
|
||||
clock_gettime 7326 199.500 0.249 0.027 0.013
|
||||
stat 2101 110.768 19.056 0.053 0.017 ENOENT: 2076
|
||||
...
|
||||
---------------
|
||||
|
||||
Parent PID: 495
|
||||
Child PIDs: 8383, 8418, 8419, 8420, 8421
|
||||
|
||||
Slowest file access times for PID 6423:
|
||||
|
||||
open (ms) timestamp error file name
|
||||
----------- --------------- --------------- ----------
|
||||
29.818 10:53:11.528954 /srv/gitlab-data/builds/2018_08/6174/954448.log
|
||||
12.309 10:53:46.708274 /srv/gitlab-data/builds/2018_08/5342/954186.log
|
||||
0.039 10:53:49.222110 /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_note.html.haml
|
||||
0.035 10:53:49.125115 /opt/gitlab/embedded/service/gitlab-rails/app/views/events/event/_push.html.haml
|
||||
...
|
||||
```
|
||||
|
||||
In the example above, we can see that file opening times on `/srv/gitlab-data` are
|
||||
extremely slow, about 100X slower than `/opt/gitlab`.
|
||||
|
||||
When nothing stands out in the results, a good way to get more context is to run `strace`
|
||||
on your own GitLab instance while performing the action performed by the customer,
|
||||
then compare summaries of both results and dive into the differences.
|
||||
|
||||
#### Stats for the open syscall
|
||||
|
||||
Rough numbers for calls to `open` and `openat` (used to access files) on various configurations.
|
||||
Slow storage can cause the dreaded `DeadlineExceeded` error in Gitaly.
|
||||
|
||||
Also [see this entry](https://docs.gitlab.com/ee/administration/operations/filesystem_benchmarking.html)
|
||||
in the handbook for quick tests customers can perform to check their filesystem performance.
|
||||
|
||||
Keep in mind that timing information from `strace` is often somewhat inaccurate, so
|
||||
small differences should not be considered significant.
|
||||
|
||||
|Setup | access times |
|
||||
|:--------------|:--------------|
|
||||
| EFS | 10 - 30ms |
|
||||
| Local Storage | 0.01 - 1ms |
|
||||
|
||||
## Networking
|
||||
|
||||
### Ports
|
||||
|
||||
```bash
|
||||
# Find the programs that are listening on ports
|
||||
netstat -plnt
|
||||
ss -plnt
|
||||
lsof -i -P | grep <port>
|
||||
```
|
||||
|
||||
### Internet/DNS
|
||||
|
||||
```bash
|
||||
# Show domain IP address
|
||||
dig +short example.com
|
||||
nslookup example.com
|
||||
|
||||
# Check DNS using specific nameserver
|
||||
# 8.8.8.8 = google, 1.1.1.1 = cloudflare, 208.67.222.222 = opendns
|
||||
dig @8.8.8.8 example.com
|
||||
nslookup example.com 1.1.1.1
|
||||
|
||||
# Find host provider
|
||||
whois <ip_address> | grep -i "orgname\|netname"
|
||||
|
||||
# Curl headers with redirect
|
||||
curl --head --location https://example.com
|
||||
```
|
||||
|
||||
## Package Management
|
||||
|
||||
```bash
|
||||
# Debian/Ubuntu
|
||||
|
||||
# List packages
|
||||
dpkg -l
|
||||
apt list --installed
|
||||
|
||||
# Find an installed package
|
||||
dpkg -l | grep <package>
|
||||
apt list --installed | grep <package>
|
||||
|
||||
# Install a package
|
||||
dpkg -i <package_name>.deb
|
||||
apt-get install <package>
|
||||
apt install <package>
|
||||
|
||||
# CentOS/RedHat
|
||||
|
||||
# Install a package
|
||||
yum install <package>
|
||||
dnf install <package> # RHEL/CentOS 8+
|
||||
|
||||
rpm -ivh <package_name>.rpm
|
||||
|
||||
# Find an installed package
|
||||
rpm -qa | grep <package>
|
||||
```
|
||||
|
||||
## Logs
|
||||
|
||||
```bash
|
||||
# Print last lines in log file where 'n'
|
||||
# is the number of lines to print
|
||||
tail -n /path/to/log/file
|
||||
```
|
|
@ -0,0 +1,126 @@
|
|||
---
|
||||
type: reference
|
||||
---
|
||||
|
||||
# Apps for a Testing Environment
|
||||
|
||||
This is the GitLab Support Team's collection of information regarding testing environments,
|
||||
for use while troubleshooting. It is listed here for transparency, and it may be useful
|
||||
for users with experience with these tools. If you are currently having an issue with
|
||||
GitLab, you may want to check your [support options](https://about.gitlab.com/support/)
|
||||
first, before attempting to use this information.
|
||||
|
||||
NOTE: **Note:**
|
||||
This page was initially written for Support Engineers, so some of the links
|
||||
are only available internally at GitLab.
|
||||
|
||||
## Docker
|
||||
|
||||
The following were tested on docker containers running in the cloud. Support Engineers,
|
||||
please see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers)
|
||||
on how to run Docker containers on `dev-resources`. Other setups haven't been tested,
|
||||
but contributions are welcome.
|
||||
|
||||
### GitLab
|
||||
|
||||
Please see [our Docker test environment docs](https://docs.gitlab.com/ee/install/digitaloceandocker.html#create-new-gitlab-container)
|
||||
for how to run GitLab on Docker. When spinning this up with `docker-machine`, ensure
|
||||
you change a few things:
|
||||
|
||||
1. Update the name of the `docker-machine` host. You can see a list of hosts
|
||||
with `docker-machine ls`.
|
||||
1. Expose the necessary ports using the `-p` flag. Docker normally doesn't
|
||||
allow access to any ports it uses outside of the container, so they must be
|
||||
explicitly exposed.
|
||||
1. Add any necessary `gitlab.rb` configuration to the
|
||||
`GITLAB_OMNIBUS_CONFIG` variable.
|
||||
|
||||
For example, when the `docker-machine` host we want to use is `do-docker`:
|
||||
|
||||
```sh
|
||||
docker run --detach --name gitlab \
|
||||
--env GITLAB_OMNIBUS_CONFIG="external_url 'http://$(docker-machine ip do-docker)'; gitlab_rails['gitlab_shell_ssh_port'] = 2222;" \
|
||||
--hostname $(docker-machine ip do-docker) \
|
||||
-p 80:80 -p 2222:22 \
|
||||
gitlab/gitlab-ee:11.5.3-ee.0
|
||||
```
|
||||
|
||||
### SAML
|
||||
|
||||
#### SAML for Authentication
|
||||
|
||||
We can use the [test-saml-idp Docker image](https://hub.docker.com/r/jamedjo/test-saml-idp)
|
||||
to do the work for us:
|
||||
|
||||
```sh
|
||||
docker run --name gitlab_saml -p 8080:8080 -p 8443:8443 \
|
||||
-e SIMPLESAMLPHP_SP_ENTITY_ID=<GITLAB_IP_OR_DOMAIN> \
|
||||
-e SIMPLESAMLPHP_SP_ASSERTION_CONSUMER_SERVICE=<GITLAB_IP_OR_DOMAIN>/users/auth/saml/callback \
|
||||
-d jamedjo/test-saml-idp
|
||||
```
|
||||
|
||||
The following will also need to go in your `/etc/gitlab/gitlab.rb`. See [our SAML docs](https://docs.gitlab.com/ee/integration/saml.html)
|
||||
for more, as well as the list of [default usernames, passwords, and emails](https://hub.docker.com/r/jamedjo/test-saml-idp/#usage).
|
||||
|
||||
```ruby
|
||||
gitlab_rails['omniauth_enabled'] = true
|
||||
gitlab_rails['omniauth_allow_single_sign_on'] = ['saml']
|
||||
gitlab_rails['omniauth_sync_email_from_provider'] = 'saml'
|
||||
gitlab_rails['omniauth_sync_profile_from_provider'] = ['saml']
|
||||
gitlab_rails['omniauth_sync_profile_attributes'] = ['email']
|
||||
gitlab_rails['omniauth_auto_sign_in_with_provider'] = 'saml'
|
||||
gitlab_rails['omniauth_block_auto_created_users'] = false
|
||||
gitlab_rails['omniauth_auto_link_ldap_user'] = false
|
||||
gitlab_rails['omniauth_auto_link_saml_user'] = true
|
||||
gitlab_rails['omniauth_providers'] = [
|
||||
{
|
||||
"name" => "saml",
|
||||
"label" => "SAML",
|
||||
"args" => {
|
||||
assertion_consumer_service_url: '<GITLAB_IP_OR_DOMAIN>/users/auth/saml/callback',
|
||||
idp_cert_fingerprint: '119b9e027959cdb7c662cfd075d9e2ef384e445f',
|
||||
idp_sso_target_url: '<SAML_IP_OR_DOMAIN>:8080/simplesaml/saml2/idp/SSOService.php',
|
||||
issuer: '<GITLAB_IP_OR_DOMAIN>',
|
||||
name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent'
|
||||
}
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### GroupSAML for GitLab.com
|
||||
|
||||
See [the GDK SAML documentation](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/saml.md).
|
||||
|
||||
### ElasticSearch
|
||||
|
||||
```sh
|
||||
docker run -d --name elasticsearch \
|
||||
-p 9200:9200 -p 9300:9300 \
|
||||
-e "discovery.type=single-node" \
|
||||
docker.elastic.co/elasticsearch/elasticsearch:5.5.1
|
||||
```
|
||||
|
||||
Then confirm it works in the browser at `curl http://<IP_ADDRESS>:9200/_cat/health`.
|
||||
ElasticSearch's default username is `elastic` and password is `changeme`.
|
||||
|
||||
### PlantUML
|
||||
|
||||
See [our PlantUML docs](../integration/plantuml.md#docker)
|
||||
on running PlantUML in Docker.
|
||||
|
||||
### Jira
|
||||
|
||||
```sh
|
||||
docker run -d -p 8081:8080 cptactionhank/atlassian-jira:latest
|
||||
```
|
||||
|
||||
Then go to `<IP_ADDRESS>:8081` in the browser to set it up. This requires a
|
||||
Jira license.
|
||||
|
||||
### Grafana
|
||||
|
||||
```sh
|
||||
docker run -d --name grafana -e "GF_SECURITY_ADMIN_PASSWORD=gitlab" -p 3000:3000 grafana/grafana
|
||||
```
|
||||
|
||||
Access it at `<IP_ADDRESS>:3000`.
|
Loading…
Reference in New Issue