gitlab-org--gitlab-foss/lib/gitlab
Yorick Peterse 41bfe82b7a
Optimise searching for users using short queries
This optimises searching for users when using queries consisting out of
one or two characters such as "ab". We optimise such cases by searching
for `LOWER(name)` and `LOWER(username)` instead of using `ILIKE`. Using
`LOWER` produces a _much_ better performing query.

For example, when searching for all users matching the term "a" we'd
produce the following plan:

     Limit  (cost=637.69..637.74 rows=20 width=805) (actual time=41.983..41.995 rows=20 loops=1)
       Buffers: shared hit=8330
       ->  Sort  (cost=637.69..638.61 rows=368 width=805) (actual time=41.982..41.990 rows=20 loops=1)
             Sort Key: (CASE WHEN ((name)::text = 'a'::text) THEN 0 WHEN ((username)::text = 'a'::text) THEN 1 WHEN ((email)::text = 'a'::text) THEN 2 ELSE 3 END), name
             Sort Method: top-N heapsort  Memory: 35kB
             Buffers: shared hit=8330
             ->  Bitmap Heap Scan on users  (cost=75.47..627.89 rows=368 width=805) (actual time=9.452..41.305 rows=277 loops=1)
                   Recheck Cond: (((name)::text ~~* 'a'::text) OR ((username)::text ~~* 'a'::text) OR ((email)::text = 'a'::text))
                   Rows Removed by Index Recheck: 7601
                   Heap Blocks: exact=7636
                   Buffers: shared hit=8327
                   ->  BitmapOr  (cost=75.47..75.47 rows=368 width=0) (actual time=8.290..8.290 rows=0 loops=1)
                         Buffers: shared hit=691
                         ->  Bitmap Index Scan on index_users_on_name_trigram  (cost=0.00..38.85 rows=180 width=0) (actual time=4.369..4.369 rows=4071 loops=1)
                               Index Cond: ((name)::text ~~* 'a'::text)
                               Buffers: shared hit=360
                         ->  Bitmap Index Scan on index_users_on_username_trigram  (cost=0.00..34.41 rows=188 width=0) (actual time=3.896..3.896 rows=4140 loops=1)
                               Index Cond: ((username)::text ~~* 'a'::text)
                               Buffers: shared hit=328
                         ->  Bitmap Index Scan on users_email_key  (cost=0.00..1.94 rows=1 width=0) (actual time=0.022..0.022 rows=0 loops=1)
                               Index Cond: ((email)::text = 'a'::text)
                               Buffers: shared hit=3
     Planning time: 3.912 ms
     Execution time: 42.171 ms

With the changes in this commit we now produce the following plan
instead:

     Limit  (cost=13257.48..13257.53 rows=20 width=805) (actual time=1.567..1.579 rows=20 loops=1)
       Buffers: shared hit=287
       ->  Sort  (cost=13257.48..13280.93 rows=9379 width=805) (actual time=1.567..1.572 rows=20 loops=1)
             Sort Key: (CASE WHEN ((name)::text = 'a'::text) THEN 0 WHEN ((username)::text = 'a'::text) THEN 1 WHEN ((email)::text = 'a'::text) THEN 2 ELSE 3 END), name
             Sort Method: top-N heapsort  Memory: 35kB
             Buffers: shared hit=287
             ->  Bitmap Heap Scan on users  (cost=135.66..13007.91 rows=9379 width=805) (actual time=0.194..1.107 rows=277 loops=1)
                   Recheck Cond: ((lower((name)::text) = 'a'::text) OR (lower((username)::text) = 'a'::text) OR ((email)::text = 'a'::text))
                   Heap Blocks: exact=277
                   Buffers: shared hit=287
                   ->  BitmapOr  (cost=135.66..135.66 rows=9379 width=0) (actual time=0.152..0.152 rows=0 loops=1)
                         Buffers: shared hit=10
                         ->  Bitmap Index Scan on yorick_test_users  (cost=0.00..124.75 rows=9377 width=0) (actual time=0.101..0.101 rows=277 loops=1)
                               Index Cond: (lower((name)::text) = 'a'::text)
                               Buffers: shared hit=4
                         ->  Bitmap Index Scan on index_on_users_lower_username  (cost=0.00..1.94 rows=1 width=0) (actual time=0.035..0.035 rows=1 loops=1)
                               Index Cond: (lower((username)::text) = 'a'::text)
                               Buffers: shared hit=3
                         ->  Bitmap Index Scan on users_email_key  (cost=0.00..1.94 rows=1 width=0) (actual time=0.014..0.014 rows=0 loops=1)
                               Index Cond: ((email)::text = 'a'::text)
                               Buffers: shared hit=3
     Planning time: 0.303 ms
     Execution time: 1.687 ms

Here we can see the new query is 25 times faster compared to the old
query.
2018-02-22 18:55:36 +01:00
..
auth Fixing request json mime type 2018-01-15 09:09:21 +00:00
background_migration Use convenient Rails helper 2018-02-21 10:04:42 -08:00
badge Change coverage badge rounding for other CI system consistency 2018-02-01 17:40:26 -06:00
bare_repository_import Look for rugged with static analysis 2018-01-25 14:05:11 +01:00
bitbucket_import
cache
checks Merge branch 'jej/avoid-slow-file-lock-checks' into 'master' 2018-02-19 15:50:54 +00:00
ci Merge branch '41209-ci-linter-fails-on-gitlab-ci-blob-viewer' into 'master' 2018-02-07 13:09:56 +00:00
conflict Simplify conflict resolution interface and code 2018-01-02 16:22:02 -03:00
cycle_analytics Remove soft removals related code 2018-01-08 17:04:45 +01:00
data_builder
database Use has_table_privilege for TRIGGER on PostgreSQL 2018-01-22 13:43:38 +01:00
dependency_linker Enable RuboCop Style/RegexpLiteral 2018-02-01 02:06:07 +09:00
diff Fix 500 error when diff context line has broken encoding 2018-02-22 12:26:23 +00:00
downtime_check
email Enable RuboCop Style/RegexpLiteral 2018-02-01 02:06:07 +09:00
etag_caching
fogbugz_import Adds Rubocop rule for line break around conditionals 2018-01-11 16:34:01 +00:00
gfm add the uploader context to the upload model 2018-02-02 09:28:15 -05:00
git Merge branch 'fix-squash-with-renamed-files' into 'master' 2018-02-22 11:35:34 +00:00
gitaly_client Incorporate Gitaly's RepositoryService.IsSquashInProgress RPC 2018-02-21 17:35:31 -03:00
github_import [GH Import] Create an empty wiki if wiki import failed 2018-02-19 17:56:33 +00:00
gitlab_import
google_code_import Adds Rubocop rule for line break around conditionals 2018-01-11 16:34:01 +00:00
gpg Fix Error 500 when viewing a commit with a GPG signature in Geo 2018-02-19 00:22:13 -08:00
grape_logging Save user ID and username in Grape API log (api_json.log) 2018-01-08 21:23:24 -08:00
graphs
health_checks Add a gRPC health check to ensure Gitaly is up 2018-01-24 15:47:27 -08:00
hook_data Remove soft removals related code 2018-01-08 17:04:45 +01:00
i18n
import_export Merge branch 'fix/template-project-visibility' into 'master' 2018-02-14 12:38:51 +00:00
kubernetes Enable Prometheus metrics for deployed Ingresses 2018-02-05 12:46:31 +00:00
ldap Fixed user synced attributes metadata after removing current provider 2018-02-15 09:27:38 +00:00
legacy_github_import use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
metrics Guard against nil Sidekiq metrics server 2018-02-20 21:47:55 -08:00
middleware Honour workhorse provided file name 2018-02-12 17:31:29 +01:00
o_auth Fixed bug with the user synced attributes when the user doesn't exist 2018-02-15 17:18:17 +01:00
performance_bar
popen Use Process::Status rather than an integer 2018-01-26 19:42:48 +08:00
project_authorizations
prometheus Make prometheus service querying approach much nicer wrt to arity and default function params 2018-02-07 01:54:09 +01:00
query_limiting Do not count rails sql cache as queries in query limiting 2018-02-21 16:27:05 +11:00
quick_actions Enable RuboCop Style/RegexpLiteral 2018-02-01 02:06:07 +09:00
redis Eliminate the last warning for redis wrapper 2018-01-26 19:42:48 +08:00
request_profiler
saml
sanitizers
serializer
sherlock Enable RuboCop Style/RegexpLiteral 2018-02-01 02:06:07 +09:00
sidekiq_middleware Explicitly set cwd in Sidekiq memory killer instead of depending on getcwd 2018-02-06 11:14:45 -06:00
sidekiq_status
sidekiq_versioning
slash_commands
sql Optimise searching for users using short queries 2018-02-22 18:55:36 +01:00
storage_check Adds Rubocop rule for line break around conditionals 2018-01-11 16:34:01 +00:00
template
testing Adds Rubocop rule for line break around conditionals 2018-01-11 16:34:01 +00:00
utils Use Gitlab::Utils::Override over defined?(super) 2017-12-26 17:18:10 +08:00
view/presenter
access.rb
action_rate_limiter.rb
allowable.rb
app_logger.rb
asciidoc.rb Set Asciidoctor outfilesuffix to default .adoc 2018-02-14 20:40:47 +00:00
auth.rb use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
background_migration.rb
blame.rb
changes_list.rb
chat_name_token.rb
ci_access.rb
closing_issue_extractor.rb
color_schemes.rb
config_helper.rb
contributions_calendar.rb
contributor.rb
current_settings.rb use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
daemon.rb
database.rb
dependency_linker.rb
downtime_check.rb
ee_compat_check.rb Ignore conflicts in db/schema.rb in Gitlab::EeCompatCheck 2018-01-30 18:22:36 +01:00
emoji.rb
encoding_helper.rb Return a warning string if we try to encode to unsupported encoding 2018-02-09 22:19:43 +01:00
environment.rb
environment_logger.rb
exclusive_lease.rb Run background migrations with a minimum interval 2018-01-05 16:23:25 +01:00
fake_application_settings.rb
file_detector.rb Enable RuboCop Style/RegexpLiteral 2018-02-01 02:06:07 +09:00
file_finder.rb API - Include project in commits&blobs search results 2018-02-14 09:00:26 +01:00
git.rb Merge branch 'gitaly-970-repo-write-config' into 'master' 2018-02-01 19:46:32 +00:00
git_access.rb Fix repo existence check in GitAccessWiki 2018-02-22 00:20:30 +00:00
git_access_wiki.rb Fix repo existence check in GitAccessWiki 2018-02-22 00:20:30 +00:00
git_logger.rb
git_post_receive.rb
git_ref_validator.rb
gitaly_client.rb Rename Concern -> Methods 2018-01-29 15:13:05 +01:00
github_import.rb
gl_id.rb
gl_repository.rb
gon_helper.rb Merge branch 'master' into jivl-update-katex 2018-02-05 15:16:41 -06:00
gpg.rb
group_hierarchy.rb
highlight.rb
i18n.rb
identifier.rb
import_export.rb Bump import/export version to 2.2.0 2018-01-05 15:18:17 +01:00
import_formatter.rb
import_sources.rb add note to import sources 2018-01-03 09:31:32 +01:00
incoming_email.rb
insecure_key_fingerprint.rb Backport authorized_keys branch 'find-key-by-fingerprint' 2018-01-08 20:34:16 +00:00
issuable_metadata.rb
issuable_sorter.rb
issuables_count_for_state.rb
issues_labels.rb
job_waiter.rb
kubernetes.rb
lazy.rb
lfs_token.rb
logger.rb
mail_room.rb
markup_helper.rb
metrics.rb Convert InfluxDB to concern. Fix uninitialized metrics when metrics code is inherited. 2018-01-29 15:13:04 +01:00
multi_collection_paginator.rb Adds Rubocop rule for line break around conditionals 2018-01-11 16:34:01 +00:00
o_auth.rb Merge branch 'jej/fix-disabled-oauth-access-10-3' into 'security-10-3' 2018-01-16 17:05:01 -08:00
optimistic_locking.rb
other_markup.rb
otp_key_rotator.rb
pages.rb
pages_transfer.rb
path_regex.rb Merge branch 'master' into '26388-push-to-create-a-new-project' 2018-02-06 19:14:43 +00:00
performance_bar.rb use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
polling_interval.rb use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
popen.rb Use Process::Status rather than an integer 2018-01-26 19:42:48 +08:00
profiler.rb Improve error handling for Gitlab::Profiler and improve doc about providing a user 2018-02-16 19:47:44 +11:00
project_search_results.rb API - Include project in commits&blobs search results 2018-02-14 09:00:26 +01:00
project_template.rb
project_transfer.rb
prometheus_client.rb Catch json parsing error as PrometheusError 2018-02-07 02:25:54 +01:00
protocol_access.rb use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
query_limiting.rb Disable query limiting warnings for now on GitLab.com 2018-02-12 11:41:03 -08:00
recaptcha.rb use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
reference_counter.rb
reference_extractor.rb
regex.rb Ensure users can't create environments with leading or trailing slashes (Fixes #39885) 2018-02-13 11:07:12 +11:00
repo_path.rb Enable RuboCop Style/RegexpLiteral 2018-02-01 02:06:07 +09:00
repository_check_logger.rb
request_context.rb
request_forgery_protection.rb
request_profiler.rb
route_map.rb
routing.rb
search_results.rb API - Include project in commits&blobs search results 2018-02-14 09:00:26 +01:00
seeder.rb Mute notifications in the seeder to speed GDK development 2018-01-26 15:25:05 -08:00
sentry.rb use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
setup_helper.rb 36847 - update toml-rb to 1.0.0 2018-02-21 22:09:15 +10:30
shell.rb Merge branch 'zj-namespace-gitaly-opt-out' into 'master' 2018-02-08 10:57:42 +00:00
shell_adapter.rb
sherlock.rb
sidekiq_config.rb
sidekiq_logger.rb
sidekiq_status.rb
sidekiq_throttler.rb
sidekiq_versioning.rb
snippet_search_results.rb Return only limited pagination headers for search API endpoints 2018-02-06 17:58:18 +01:00
ssh_public_key.rb Validate SSH keys through the sshkey gem 2018-02-16 11:32:08 -05:00
storage_check.rb
string_range_marker.rb
string_regex_marker.rb
task_helpers.rb Fix rubocop offenses. It's not checked before when 2018-01-26 19:42:48 +08:00
tcp_checker.rb
themes.rb
time_tracking_formatter.rb
timeless.rb Adds Rubocop rule for line break around conditionals 2018-01-11 16:34:01 +00:00
untrusted_regexp.rb
update_path_error.rb
upgrader.rb Enable RuboCop Style/RegexpLiteral 2018-02-01 02:06:07 +09:00
uploads_transfer.rb port of 594e6a0a625^..f74c90f68c6 2018-02-01 12:14:46 -05:00
url_blocker.rb
url_builder.rb
url_sanitizer.rb
usage_data.rb use Gitlab::UserSettings directly as a singleton instead of including/extending it 2018-02-02 18:39:55 +00:00
user_access.rb Moves project creationg to git access check for git push 2018-02-06 16:52:29 +00:00
user_activities.rb
utils.rb Merge branch '41293-fix-command-injection-vulnerability-on-system_hook_push-queue-through-web-hook' into 'security-10-3' 2018-01-16 17:04:51 -08:00
version_info.rb
visibility_level.rb Merge branch 'bvl-fix-500-on-fork-without-restricted-visibility-levels' into 'master' 2018-02-02 19:12:55 +00:00
workhorse.rb Upgrade GitLab Workhorse to v3.6.0 2018-02-06 15:34:44 +01:00