Commit graph

17 commits

Author SHA1 Message Date
Stan Hu
038be9fffa Fix Error 500s due to encoding issues when Wiki hooks fire
Saved Wiki content goes through the GitalyClient::WikiService, which calls
StringIO#set_encoding on the input stream. The problem is that this call
mutates the encoding of the given string object to ASCII-88BIT, which
causes problems for models expecting the data to still be in UTF-8.

Freezing the input disables this behavior:
https://github.com/ruby/ruby/blob/v2_4_4/ext/stringio/stringio.c#L1583

Closes #50590
2018-08-29 09:46:46 -07:00
Lin Jen-Shin
3bfe306624 Resolve Naming/UncommunicativeMethod 2018-07-09 21:13:08 +08:00
Lin Jen-Shin
4ee08b77bc Updates from rubocop -a 2018-07-09 21:13:08 +08:00
Ahmad Sherif
1f2cc29c8f Fix EncodingHelper#clean blowing up on UTF-16BE strings
Closes gitaly#1101
2018-03-22 21:22:20 +01:00
Ahmad Sherif
0d9107374a Return a warning string if we try to encode to unsupported encoding
Fixes gitlab-development-kit#321
2018-02-09 22:19:43 +01:00
Alejandro Rodríguez
5152cc3bfb Fix a bug where charlock_holmes was used needlessly to encode strings 2018-01-04 19:27:37 -03:00
Alejandro Rodríguez
11247ac97c Move encoding methods to the more general EncodingHelper 2017-12-28 12:32:33 -03:00
Jacob Vosmaer (GitLab)
4cfcc97544 Fix encoding bugs in Gitlab::Git::User 2017-11-23 10:48:57 +00:00
Alejandro Rodríguez
520866a0d0 Avoind unnecesary force_encoding operations
They're costly. This will also avoid some edge cases where
charlock_holmes assigns a weird encoding to a perfectly valid UTF-8
string.
2017-10-18 21:30:29 -03:00
micael.bergeron
c1f66cc7f0 remove useless comment 2017-09-06 09:15:27 -04:00
micael.bergeron
a6af5522d7 renames ambiguous methods and add spec 2017-09-06 09:01:53 -04:00
micael.bergeron
dbaed90c8d fix refactoring error with Blob.binary?
remove some lint
2017-09-06 09:01:53 -04:00
micael.bergeron
c9aa793215 revert to using a simple representation 2017-09-06 09:01:53 -04:00
micael.bergeron
bca72f5906 wip: fake its a binary diff 2017-09-06 09:01:53 -04:00
Lin Jen-Shin
8e2350ae95 Raise encoding confidence threshold to 50
It is recommended that we set this to 50:
https://gitlab.com/gitlab-org/gitlab-ce/issues/35098#note_35036746

In this particular issue, the confidence was 42 for Shift JIS,
but in fact that's encoded in UTF-8 just with a single bad
character. In this case, we shouldn't try to treat it as Shift JIS,
but just treat it as UTF-8 and remove invalid bytes.

Treating it like Shift JIS would corrupt the whole data.

Unfortunately, the diff which would cause this could not be
disclosed therefore we can't use it as a test example.
2017-08-08 19:09:56 +08:00
James Lopez
6b53add3f9 Fix binary encoding error on MR diffs 2017-06-06 16:40:07 +00:00
Bob Van Landuyt
e564fe971f Rename Gitlab::Git::EncodingHelper to Gitlab::EncodingHelper 2017-06-01 21:21:14 +00:00