Pre-calculate Emoji digests

By pre-calculating the digests we can manually construct the emoji URLs,
removing the need for using Rails' asset URL helpers. The reason we
don't want to use these helpers for Emojis is two-fold:

1. Rails' image_url() method is slow, really slow. For one it _might_
   have to calculate digests but it also performs a lot of other
   intensive operations (judging by the source code and based on
   measuring timings).

2. We have a lot of Emoji which coupled with the above can result in it
   taking minutes to load Emoji autocomplete data.

Using this pre-calculation setup generating the digests takes around 7
seconds (including the time it takes to start Rails/Rake), and only
around 600 milliseconds to load _all_ the autocomplete data of a project
(measured locally).

This commit _does_ change the Emoji URLs from absolute to relative URLs
as these are much easier to generate.

To update the Emoji data simply run:

    rake gemojione:digests

Then commit any changes.

Fixes gitlab-org/gitlab-ce#14009
This commit is contained in:
Yorick Peterse 2016-03-30 19:29:17 +02:00
parent 8718acdf80
commit 5830d80b8d
5 changed files with 8670 additions and 26 deletions

View file

@ -138,7 +138,7 @@ class ProjectsController < Projects::ApplicationController
participants = ::Projects::ParticipantsService.new(@project, current_user).execute(note_type, note_id)
@suggestions = {
emojis: autocomplete_emojis,
emojis: AwardEmoji.urls,
issues: autocomplete.issues,
mergerequests: autocomplete.merge_requests,
members: participants
@ -235,17 +235,6 @@ class ProjectsController < Projects::ApplicationController
)
end
def autocomplete_emojis
Rails.cache.fetch("autocomplete-emoji-#{Gemojione::VERSION}") do
Emoji.emojis.map do |name, emoji|
{
name: name,
path: view_context.image_url("#{emoji["unicode"]}.png")
}
end
end
end
def repo_exists?
project.repository_exists? && !project.empty_repo?
end

8597
fixtures/emojis/digests.json Normal file

File diff suppressed because it is too large Load diff

View file

@ -48,4 +48,23 @@ class AwardEmoji
JSON.parse(File.read(json_path))
end
end
# Returns an Array of Emoji names and their asset URLs.
def self.urls
@urls ||= begin
path = File.join(Rails.root, 'fixtures', 'emojis', 'digests.json')
prefix = Gitlab::Application.config.assets.prefix
digest = Gitlab::Application.config.assets.digest
JSON.parse(File.read(path)).map do |hash|
if digest
fname = "#{hash['unicode']}-#{hash['digest']}"
else
fname = hash['unicode']
end
{ name: hash['name'], path: "#{prefix}/#{fname}.png" }
end
end
end
end

View file

@ -1,19 +1,39 @@
# This task will generate a standard and Retina sprite of all of the current
# Gemojione Emojis, with the accompanying SCSS map.
#
# It will not appear in `rake -T` output, and the dependent gems are not
# included in the Gemfile by default, because this task will only be needed
# occasionally, such as when new Emojis are added to Gemojione.
begin
require 'sprite_factory'
require 'rmagick'
rescue LoadError
# noop
end
namespace :gemojione do
desc 'Generates Emoji SHA256 digests'
task digests: :environment do
require 'digest/sha2'
require 'json'
dir = Gemojione.index.images_path
digests = AwardEmoji.emojis.map do |name, emoji_hash|
fpath = File.join(dir, "#{emoji_hash['unicode']}.png")
digest = Digest::SHA256.file(fpath).hexdigest
{ name: name, unicode: emoji_hash['unicode'], digest: digest }
end
out = File.join(Rails.root, 'fixtures', 'emojis', 'digests.json')
File.open(out, 'w') do |handle|
handle.write(JSON.pretty_generate(digests))
end
end
# This task will generate a standard and Retina sprite of all of the current
# Gemojione Emojis, with the accompanying SCSS map.
#
# It will not appear in `rake -T` output, and the dependent gems are not
# included in the Gemfile by default, because this task will only be needed
# occasionally, such as when new Emojis are added to Gemojione.
task sprite: :environment do
begin
require 'sprite_factory'
require 'rmagick'
rescue LoadError
# noop
end
check_requirements!
SIZE = 20

View file

@ -0,0 +1,19 @@
require 'spec_helper'
describe AwardEmoji do
describe '.urls' do
subject { AwardEmoji.urls }
it { is_expected.to be_an_instance_of(Array) }
it { is_expected.to_not be_empty }
context 'every Hash in the Array' do
it 'has the correct keys and values' do
subject.each do |hash|
expect(hash[:name]).to be_an_instance_of(String)
expect(hash[:path]).to be_an_instance_of(String)
end
end
end
end
end