gitlab-org--gitlab-foss/lib/gitlab
Kamil Trzciński 0e56c1e7cb Improve performance and memory usage of project export
ActiveModel::Serialization is simple in that it recursively calls
`as_json` on each object to serialize everything. However, for a model
like a Project, this can generate a query for every single association,
which can add up to tens of thousands of queries and lead to memory
bloat.

To improve this, we can do several things:

1. We use `tree:` and `preload:` to automatically generate
   a list of all preloads that could be used to serialize
   objects in bulk.

2. We observe that a single project has many issues, merge requests,
   etc. Instead of serializing everything at once, which could lead to
   database timeouts and high memory usage, we take each top-level
   association and serialize the data in batches.

For example, we serialize the first 100 issues and preload all of
their associated events, notes, etc. before moving onto the next
batch. When we're done, we serialize merge requests in the same way.
We repeat this pattern for the remaining associations specified in
import_export.yml.
2019-09-09 15:40:49 +00:00
..
access
action_view_output
analytics/cycle_analytics
asciidoc
auth
background_migration
badge
bare_repository_import
bitbucket_import
bitbucket_server_import
cache
chat
checks
ci
cleanup
cluster
config
conflict
content_security_policy
cross_project_access
cycle_analytics
danger
data_builder
database
database_importers
dependency_linker
diff
discussions_diff
downtime_check
email
error_tracking
etag_caching
external_authorization
fogbugz_import
gfm
git
git_access_result
gitaly_client
github_import
gitlab_import
gl_repository
google_code_import
gpg
grape_logging
graphql
graphs
hashed_storage
health_checks
hook_data
i18n
import
import_export Improve performance and memory usage of project export 2019-09-09 15:40:49 +00:00
internal_post_receive
jira
kubernetes
legacy_github_import
lets_encrypt
manifest_import
markdown_cache
metrics
middleware
octokit
patch
performance_bar
phabricator_import
popen
profiler
prometheus
query_limiting
quick_actions
redis
request_profiler
sanitizers
search
serializer
sherlock
sidekiq_logging
sidekiq_middleware Fix for histogram corruption in Sidekiq 2019-08-30 14:04:45 +00:00
sidekiq_status
sidekiq_versioning
slash_commands
slug
sql
template
testing
untrusted_regexp
usage_data_counters
utils
verify
view/presenter
webpack
access.rb
action_rate_limiter.rb
allowable.rb
anonymous_session.rb
app_logger.rb
asciidoc.rb
audit_json_logger.rb
auth.rb
auth_logger.rb
authorized_keys.rb
background_migration.rb
base_doorkeeper_controller.rb
batch_pop_queueing.rb
blame.rb
blob_helper.rb
branch_push_merge_commit_analyzer.rb
build_access.rb
changes_list.rb
chaos.rb
chat.rb
chat_name_token.rb
ci_access.rb
closing_issue_extractor.rb
color_schemes.rb
config_helper.rb
content_disposition.rb
contributions_calendar.rb
contributor.rb
cross_project_access.rb
crypto_helper.rb
current_settings.rb
daemon.rb
database.rb
dependency_linker.rb
downtime_check.rb
ee_compat_check.rb
emoji.rb
encoding_helper.rb
environment.rb
environment_logger.rb
exclusive_lease.rb
exclusive_lease_helpers.rb
external_authorization.rb
fake_application_settings.rb
favicon.rb
file_detector.rb
file_finder.rb
file_markdown_link_builder.rb
file_type_detection.rb
git.rb
git_access.rb
git_access_wiki.rb
git_logger.rb
git_post_receive.rb
git_ref_validator.rb
gitaly_client.rb
github_import.rb
gl_id.rb
gl_repository.rb
global_id.rb
gon_helper.rb
gpg.rb
graphql.rb
graphql_logger.rb
group_search_results.rb
highlight.rb
http.rb
http_connection_adapter.rb
http_io.rb
i18n.rb
identifier.rb
import_export.rb
import_formatter.rb
import_sources.rb
incoming_email.rb
insecure_key_fingerprint.rb
instrumentation_helper.rb
issuable_metadata.rb
issuable_sorter.rb
issuables_count_for_state.rb
issues_labels.rb
job_waiter.rb
json_cache.rb
json_logger.rb
jwt_authenticatable.rb
kubernetes.rb
language_data.rb
language_detection.rb
lazy.rb
lets_encrypt.rb
lfs_token.rb
logger.rb
loop_helpers.rb
mail_room.rb
markdown_cache.rb
markup_helper.rb
metrics.rb
multi_collection_paginator.rb
namespace_sanitizer.rb
namespaced_session_store.rb
null_request_store.rb
object_hierarchy.rb
omniauth_initializer.rb
optimistic_locking.rb
other_markup.rb
otp_key_rotator.rb
pages.rb
pages_client.rb
pages_transfer.rb
path_regex.rb
performance_bar.rb
phabricator_import.rb
plugin.rb
plugin_logger.rb
polling_interval.rb
popen.rb
private_commit_email.rb
profiler.rb
project_authorizations.rb
project_search_results.rb
project_service_logger.rb
project_template.rb
project_transfer.rb
prometheus_client.rb
protocol_access.rb
push_options.rb
query_limiting.rb
recaptcha.rb
reference_counter.rb
reference_extractor.rb
regex.rb
repo_path.rb
repository_cache.rb
repository_cache_adapter.rb
repository_check_logger.rb
request_context.rb
request_forgery_protection.rb
request_profiler.rb
route_map.rb
routing.rb
rugged_instrumentation.rb
safe_request_store.rb
search_results.rb
seeder.rb
sentry.rb
session.rb
setup_helper.rb
shard_health_cache.rb
shell.rb
shell_adapter.rb
sherlock.rb
sidekiq_config.rb
sidekiq_logger.rb
sidekiq_monitor.rb
sidekiq_signals.rb
sidekiq_status.rb
sidekiq_versioning.rb
snippet_search_results.rb
ssh_public_key.rb
string_placeholder_replacer.rb
string_range_marker.rb
string_regex_marker.rb
submodule_links.rb
task_helpers.rb
tcp_checker.rb
template_helper.rb
temporarily_allow.rb
themes.rb
thread_memory_cache.rb
time_tracking_formatter.rb
timeless.rb
tracing.rb
tracking.rb
tree_summary.rb
untrusted_regexp.rb
update_path_error.rb
uploads_transfer.rb
url_blocker.rb
url_builder.rb
url_helpers.rb
url_sanitizer.rb
usage_data.rb
user_access.rb
utils.rb
version_info.rb
visibility_level.rb
visibility_level_checker.rb
wiki_file_finder.rb
workhorse.rb
zoom_link_extractor.rb