gitlab-org--gitlab-foss/lib/gitlab
Kamil Trzciński 0e56c1e7cb Improve performance and memory usage of project export
ActiveModel::Serialization is simple in that it recursively calls
`as_json` on each object to serialize everything. However, for a model
like a Project, this can generate a query for every single association,
which can add up to tens of thousands of queries and lead to memory
bloat.

To improve this, we can do several things:

1. We use `tree:` and `preload:` to automatically generate
   a list of all preloads that could be used to serialize
   objects in bulk.

2. We observe that a single project has many issues, merge requests,
   etc. Instead of serializing everything at once, which could lead to
   database timeouts and high memory usage, we take each top-level
   association and serialize the data in batches.

For example, we serialize the first 100 issues and preload all of
their associated events, notes, etc. before moving onto the next
batch. When we're done, we serialize merge requests in the same way.
We repeat this pattern for the remaining associations specified in
import_export.yml.
2019-09-09 15:40:49 +00:00
..
access
action_view_output Fix slow performance with compiling HAML templates 2019-04-29 05:33:50 -07:00
analytics/cycle_analytics Implement validation logic to ProjectStage 2019-08-23 20:28:11 +00:00
asciidoc Enable AsciiDoc syntax highlighting (using Rouge) 2019-07-02 09:24:05 +02:00
auth Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
background_migration Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
badge Rename latest_successful to be more explicit 2019-07-26 00:17:52 +02:00
bare_repository_import Adds the Rubocop ReturnNil cop 2019-03-06 17:51:56 +02:00
bitbucket_import Add code review suggestions 2019-08-12 15:05:49 +01:00
bitbucket_server_import Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
cache
chat Move ChatOps to Core 2019-02-20 21:29:48 +00:00
checks Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
ci Passing job:rules downstream and E2E specs for job:rules configuration 2019-09-09 13:21:26 +00:00
cleanup Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
cluster Remove worker label from puma terminations metric 2019-08-12 13:52:15 +03:00
config Introducing new Syntax for Ci::Build inclusion rules 2019-08-20 20:03:43 +00:00
conflict
content_security_policy Add missing report-uri to CSP config 2019-08-07 11:21:08 -07:00
cross_project_access
cycle_analytics Fix error on project name 2019-08-05 21:15:00 +00:00
danger Add new GitlabDanger class 2019-09-06 11:21:53 +00:00
data_builder Fix pipelines not always being created after a push 2019-08-17 06:23:26 -07:00
database Refactor new undo_* methods 2019-09-05 12:15:16 +00:00
database_importers Handle :9090 and 0.0.0.0:9090 listen_address formats 2019-09-02 14:35:02 +00:00
dependency_linker Raise not implemented error on BaseLinker for package_url 2019-02-25 10:22:13 -03:00
diff Fixing #65389 2019-09-02 10:43:54 +00:00
discussions_diff Remove cleaned up OIDs from database and cache 2019-05-06 11:35:03 +01:00
downtime_check Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
email feat: SMIME signed notification emails 2019-08-20 16:13:32 +02:00
error_tracking
etag_caching Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
external_authorization Adds identity information while making external authorization requests 2019-06-19 06:04:33 +00:00
fogbugz_import Change default visibility level for FogBugz imported projects to Private 2019-08-26 16:19:47 +00:00
gfm Prevent rewritting plain links as embedded 2019-08-07 15:39:48 -03:00
git Merge branch 'remove-unnecessary-freeze-in-lib-gitlab' into 'master' 2019-09-05 08:16:24 +00:00
git_access_result Allow console messages be sent to gitlab-shell 2019-04-04 14:20:11 +11:00
gitaly_client Update Gitaly server and gem to 1.58.0 2019-08-12 15:50:30 +00:00
github_import Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
gitlab_import
gl_repository Make Gitlab::GlRepository#types an instance method 2019-04-30 17:17:08 +02:00
google_code_import
gpg Fix GPG signature verification with recent versions of GnuPG 2019-06-17 13:33:39 +00:00
grape_logging Standardize remote_ip and path keys for auth.log and api_json.log 2019-08-20 18:12:28 +00:00
graphql Upgrade graphql gem to 1.9.10 2019-09-04 21:57:37 +00:00
graphs
hashed_storage Display the amount for Hashed Storage migration/rollback correctly 2019-07-10 22:14:50 +01:00
health_checks Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
hook_data Added labels_hook_attrs method 2019-06-24 11:51:34 +02:00
i18n Adds the Rubocop ReturnNil cop 2019-03-06 17:51:56 +02:00
import Remove dead MySQL code 2019-07-23 16:53:03 +01:00
import_export Improve performance and memory usage of project export 2019-09-09 15:40:49 +00:00
internal_post_receive Simplify internal post receive messages 2019-08-28 15:18:58 -07:00
jira Fix DNS rebind vulnerability for JIRA integration 2019-08-08 10:24:43 -03:00
kubernetes Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
legacy_github_import Do not allow localhost url redirection in GitHub Integration 2019-07-05 15:09:04 +05:30
lets_encrypt Use project depended feature flag for pages ssl 2019-06-25 10:30:12 +02:00
manifest_import
markdown_cache Banzai - avoid redis if attr is in DB cache 2019-07-10 21:35:43 -06:00
metrics Remove feature flags starting with `prometheus_transaction_` 2019-09-05 11:01:02 +02:00
middleware Process workhorse accelerated wiki uploads 2019-09-06 15:53:13 +02:00
octokit Refactor SystemHookUrlValidator and specs 2019-08-02 15:39:18 +01:00
patch Fix time tracking parsing of months 2019-09-05 20:24:33 +08:00
performance_bar Merge branch 'fix-peek-on-puma' into 'master' 2019-09-02 08:43:21 +00:00
phabricator_import Fetch users from Phabricator to link to issues 2019-07-10 17:15:43 +02:00
popen
profiler
prometheus Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
query_limiting Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
quick_actions Only show `/copy_metadata` when usable 2019-09-06 16:16:39 +00:00
redis Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
request_profiler Bring backward compatibility for request profiles 2019-07-23 09:30:00 +00:00
sanitizers Add direct upload support for personal snippets 2019-08-23 11:19:14 +02:00
search Fix a typo in lib/gitlab/search/found_blob.rb 2019-07-01 10:43:06 +02:00
serializer
sherlock Further remove code branches by database type 2019-07-29 12:47:06 +02:00
sidekiq_logging Log time spent on CPU to sidekiq.log 2019-08-22 14:31:57 +00:00
sidekiq_middleware Fix for histogram corruption in Sidekiq 2019-08-30 14:04:45 +00:00
sidekiq_status Rework `Sidekiq::JobsThreads` into `Monitor` 2019-08-21 12:05:30 +02:00
sidekiq_versioning
slash_commands Tidy up conditional message 2019-09-02 17:52:11 +12:00
slug Introduce predictable environment slugs 2019-07-19 11:33:07 +10:00
sql Ignore min_chars_for_partial_matching unles trigrm 2019-07-04 10:56:13 -06:00
template Merge branch 'require-all-templates-to-include-default-stages' into 'master' 2019-04-15 12:47:35 +02:00
testing
untrusted_regexp Allow to use untrusted Regexp via feature flag 2019-04-04 15:00:56 +00:00
usage_data_counters Add usage pings for merge request creating 2019-08-26 14:30:45 +00:00
utils Extract SanitizeNodeLink and apply to WikiLinkFilter 2019-07-26 13:41:11 +00:00
verify
view/presenter
webpack
access.rb Add "allowed to create subgroups" dropdown to group settings form 2019-07-19 11:55:46 -07:00
action_rate_limiter.rb Standardize remote_ip and path keys for auth.log and api_json.log 2019-08-20 18:12:28 +00:00
allowable.rb
anonymous_session.rb Add captcha if there are multiple failed login attempts 2019-07-31 11:47:55 +02:00
app_logger.rb
asciidoc.rb Enable section anchors 2019-07-16 12:53:38 +02:00
audit_json_logger.rb
auth.rb If user can push to docker then it can delete too 2019-09-03 02:41:22 +00:00
auth_logger.rb Changes RackAttack logger to use structured logs 2019-05-24 00:45:02 +00:00
authorized_keys.rb Auto create authorized_keys file if doesn't exist 2019-08-29 16:33:04 +08:00
background_migration.rb Add methods to check dead and retrying jobs 2019-04-08 19:47:36 -03:00
base_doorkeeper_controller.rb
batch_pop_queueing.rb Efficient merge train locks 2019-07-11 13:44:12 +07:00
blame.rb
blob_helper.rb Fix inline rendering of SVGs from current repo 2019-08-02 06:34:20 +08:00
branch_push_merge_commit_analyzer.rb
build_access.rb
changes_list.rb
chaos.rb Adds chaos endpoints to Sidekiq 2019-07-18 19:04:12 +02:00
chat.rb Move ChatOps to Core 2019-02-20 21:29:48 +00:00
chat_name_token.rb
ci_access.rb
closing_issue_extractor.rb
color_schemes.rb
config_helper.rb
content_disposition.rb Run rubocop -a on CE files 2019-05-05 03:24:28 -07:00
contributions_calendar.rb Further remove code branches by database type 2019-07-29 12:47:06 +02:00
contributor.rb
cross_project_access.rb
crypto_helper.rb
current_settings.rb Save instance administration project id in DB 2019-08-07 18:40:36 +00:00
daemon.rb Improve resillency of monitor 2019-08-21 12:06:10 +02:00
database.rb Merge branch 'master' of dev.gitlab.org:gitlab/gitlabhq 2019-08-29 17:17:37 -05:00
dependency_linker.rb
downtime_check.rb
ee_compat_check.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
emoji.rb
encoding_helper.rb Add a rubocop for Rails.logger 2019-07-10 19:26:47 +00:00
environment.rb
environment_logger.rb
exclusive_lease.rb
exclusive_lease_helpers.rb Add exclusive lease to mergeability check process 2019-07-31 19:58:43 -03:00
external_authorization.rb Move Contribution Analytics related spec in spec/features/groups/group_page_with_external_authorization_service_spec to EE 2019-04-09 15:38:58 +00:00
fake_application_settings.rb Introduce ApplicationSettingImplementation yay 2019-03-19 13:01:37 +08:00
favicon.rb fix(favicon): get favicon_path, so it works also with uploads object store 2019-06-14 13:58:51 +02:00
file_detector.rb Update metrics dashboard API to load yml from repo 2019-05-01 10:16:03 +00:00
file_finder.rb
file_markdown_link_builder.rb
file_type_detection.rb
git.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
git_access.rb Port changes for design management to CE 2019-04-29 08:28:43 +02:00
git_access_wiki.rb
git_logger.rb Rename githost.log -> git_json.log 2019-08-08 13:39:11 -07:00
git_post_receive.rb Expire project caches once per push instead of once per ref 2019-08-16 19:53:56 +00:00
git_ref_validator.rb Validate MR branch names 2019-05-03 03:02:58 +08:00
gitaly_client.rb Merge branch 'fix-peek-on-puma' into 'master' 2019-09-02 08:43:21 +00:00
github_import.rb
gl_id.rb
gl_repository.rb Make Gitlab::GlRepository#types an instance method 2019-04-30 17:17:08 +02:00
global_id.rb GraphQL support for Notes created in discussions 2019-07-10 12:13:48 +12:00
gon_helper.rb Remove gfm_embed_metrics flag from BE 2019-08-09 20:35:43 +00:00
gpg.rb
graphql.rb Remove `:graphql` feature flag 2019-07-09 12:45:23 +00:00
graphql_logger.rb Implement logger analyzer 2019-05-30 18:27:28 +12:00
group_search_results.rb Add improvements to the global search process 2019-05-07 11:08:25 +00:00
highlight.rb Revert "Merge branch '65152-selective-highlight' into 'master'" 2019-08-09 00:13:09 +00:00
http.rb Add OpenSSL::OpenSSLError to HTTP_ERRORS 2019-07-05 21:05:24 +00:00
http_connection_adapter.rb Add outbound requests setting for system hooks 2019-08-02 15:39:18 +01:00
http_io.rb
i18n.rb
identifier.rb
import_export.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
import_formatter.rb
import_sources.rb Setup Phabricator import 2019-05-31 09:40:54 +02:00
incoming_email.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
insecure_key_fingerprint.rb
instrumentation_helper.rb Add Gitaly and Rugged call timing in Sidekiq logs 2019-08-09 01:08:32 -07:00
issuable_metadata.rb Expose merge requests count based on user access 2019-06-18 12:46:46 +03:00
issuable_sorter.rb
issuables_count_for_state.rb
issues_labels.rb
job_waiter.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
json_cache.rb Parse the cached value when the it is false 2019-06-25 19:30:05 -03:00
json_logger.rb Migrate correlation and tracing code to LabKit 2019-04-18 09:57:16 +02:00
jwt_authenticatable.rb Extract Workhorse <-> GitLab authentication to make it reusable 2019-09-04 11:03:20 +12:00
kubernetes.rb Select deployments that only has the app label 2019-06-21 20:45:06 +00:00
language_data.rb
language_detection.rb
lazy.rb
lets_encrypt.rb Remove auto ssl feature flags 2019-07-12 16:53:44 +00:00
lfs_token.rb Geo: Remove Gitlab::LfsToken::LegacyRedisDeviseToken implementation 2019-05-22 16:11:08 +10:00
logger.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
loop_helpers.rb
mail_room.rb
markdown_cache.rb Re-escape whole HTML content instead of only match 2019-08-23 10:36:51 +02:00
markup_helper.rb
metrics.rb
multi_collection_paginator.rb
namespace_sanitizer.rb
namespaced_session_store.rb CE port of "Require session with smartcard login for Git access" 2019-07-05 08:12:29 +02:00
null_request_store.rb
object_hierarchy.rb Remove code related to object hierarchy in MySQL 2019-07-25 15:35:06 +08:00
omniauth_initializer.rb Symbolize client_auth_method for quirked providers 2019-07-15 09:01:55 -05:00
optimistic_locking.rb Backport EE issue 12996 2019-09-02 12:59:30 +02:00
other_markup.rb
otp_key_rotator.rb
pages.rb Add skeleton Pages internal API 2019-09-06 16:06:25 +12:00
pages_client.rb Add a rubocop for Rails.logger 2019-07-10 19:26:47 +00:00
pages_transfer.rb
path_regex.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
performance_bar.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
phabricator_import.rb Setup Phabricator import 2019-05-31 09:40:54 +02:00
plugin.rb
plugin_logger.rb
polling_interval.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
popen.rb
private_commit_email.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
profiler.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
project_authorizations.rb Remove code related to object hierarchy in MySQL 2019-07-25 15:35:06 +08:00
project_search_results.rb Load search result counts asynchronously 2019-08-12 22:01:15 +02:00
project_service_logger.rb
project_template.rb Squash project templates on update 2019-08-15 20:20:08 +00:00
project_transfer.rb
prometheus_client.rb Convert RestClient to Gitlab::HTTP for Prometheus Monitor 2019-08-07 02:42:20 +00:00
protocol_access.rb
push_options.rb Refactor parse_options() in push_options.rb 2019-09-07 17:31:23 +02:00
query_limiting.rb
recaptcha.rb Add captcha if there are multiple failed login attempts 2019-07-31 11:47:55 +02:00
reference_counter.rb Add a rubocop for Rails.logger 2019-07-10 19:26:47 +00:00
reference_extractor.rb
regex.rb Fixing #65389 2019-09-02 10:43:54 +00:00
repo_path.rb Fall back to project repository type by default 2019-04-05 10:51:42 +00:00
repository_cache.rb
repository_cache_adapter.rb Revert "Cache branch and tag names as Redis sets" 2019-08-29 18:04:52 +01:00
repository_check_logger.rb
request_context.rb Fix health checks not working behind load balancers 2019-03-12 12:46:40 -07:00
request_forgery_protection.rb
request_profiler.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
route_map.rb
routing.rb Fix typos in the whole gitlab-ce project 2019-05-20 14:11:44 +00:00
rugged_instrumentation.rb Make performance bar enabled checks consistent 2019-08-28 17:25:02 +01:00
safe_request_store.rb
search_results.rb Lower searches count limit 2019-09-09 13:46:24 +00:00
seeder.rb
sentry.rb Add support for sentry_extra_data in exceptions 2019-08-19 23:06:21 +00:00
session.rb Session stored globally per request 2019-05-02 12:28:26 +07:00
setup_helper.rb Only use Gitaly catfile_cache_size=5 in test 2019-06-06 14:11:44 +00:00
shard_health_cache.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
shell.rb Remove the fallback path from gitlab-ce 2019-08-29 16:33:04 +08:00
shell_adapter.rb
sherlock.rb
sidekiq_config.rb Change path of bin/sidkiq-cluster in comments 2019-04-29 10:32:33 +10:00
sidekiq_logger.rb
sidekiq_monitor.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
sidekiq_signals.rb Fix Sidekiq process group killing 2019-03-07 12:37:59 +00:00
sidekiq_status.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
sidekiq_versioning.rb
snippet_search_results.rb Load search result counts asynchronously 2019-08-12 22:01:15 +02:00
ssh_public_key.rb
string_placeholder_replacer.rb
string_range_marker.rb
string_regex_marker.rb
submodule_links.rb Fix error rendering submodules in MR diffs when there is no .gitmodules 2019-07-26 11:05:56 +01:00
task_helpers.rb
tcp_checker.rb
template_helper.rb
temporarily_allow.rb
themes.rb
thread_memory_cache.rb Add a memory cache local to the thread to reduce Redis load 2019-07-01 22:23:01 -07:00
time_tracking_formatter.rb Rename to time_tracking_limit_to_hours 2019-06-25 09:31:23 +08:00
timeless.rb
tracing.rb Remove all references to Gitlab::CorrelationId 2019-08-26 18:49:58 +02:00
tracking.rb Changes snowplow to use cookies for sessions 2019-08-28 06:52:14 +00:00
tree_summary.rb Adds the Rubocop ReturnNil cop 2019-03-06 17:51:56 +02:00
untrusted_regexp.rb Bring back Gitlab::UntrustedRegexp.with_fallback 2019-04-03 15:31:56 +08:00
update_path_error.rb
uploads_transfer.rb
url_blocker.rb Allow not resolvable urls when rebinding setting is disabled 2019-09-05 06:07:17 +00:00
url_builder.rb Update deployment chat message notification 2019-05-02 16:07:26 +00:00
url_helpers.rb Avoid calling freeze on already frozen strings in lib/gitlab 2019-09-04 09:52:02 +05:30
url_sanitizer.rb Hide password on import by url form 2019-05-29 14:03:50 +03:00
usage_data.rb Add usage pings for merge request creating 2019-08-26 14:30:45 +00:00
user_access.rb Add documentation and tests 2019-06-19 07:08:56 +00:00
utils.rb Frozen string cannot change encoding 2019-07-26 00:13:25 +12:00
version_info.rb
visibility_level.rb Fix inability to set visibility_level on project via API 2019-06-14 07:36:35 -07:00
visibility_level_checker.rb Fix project import restricted visibility bypass 2019-08-15 10:30:47 +01:00
wiki_file_finder.rb
workhorse.rb Merge branch 'remove-unnecessary-freeze-in-lib-gitlab' into 'master' 2019-09-05 08:16:24 +00:00
zoom_link_extractor.rb Add system notes for when a zoom call was added/removed from an issue 2019-07-29 21:51:16 +02:00