Ahmad Sherif
1f2cc29c8f
Fix EncodingHelper#clean blowing up on UTF-16BE strings
...
Closes gitaly#1101
2018-03-22 21:22:20 +01:00
Ahmad Sherif
0d9107374a
Return a warning string if we try to encode to unsupported encoding
...
Fixes gitlab-development-kit#321
2018-02-09 22:19:43 +01:00
Alejandro Rodríguez
5152cc3bfb
Fix a bug where charlock_holmes was used needlessly to encode strings
2018-01-04 19:27:37 -03:00
Alejandro Rodríguez
11247ac97c
Move encoding methods to the more general EncodingHelper
2017-12-28 12:32:33 -03:00
Jacob Vosmaer (GitLab)
4cfcc97544
Fix encoding bugs in Gitlab::Git::User
2017-11-23 10:48:57 +00:00
Alejandro Rodríguez
520866a0d0
Avoind unnecesary force_encoding operations
...
They're costly. This will also avoid some edge cases where
charlock_holmes assigns a weird encoding to a perfectly valid UTF-8
string.
2017-10-18 21:30:29 -03:00
micael.bergeron
c1f66cc7f0
remove useless comment
2017-09-06 09:15:27 -04:00
micael.bergeron
a6af5522d7
renames ambiguous methods and add spec
2017-09-06 09:01:53 -04:00
micael.bergeron
dbaed90c8d
fix refactoring error with Blob.binary?
...
remove some lint
2017-09-06 09:01:53 -04:00
micael.bergeron
c9aa793215
revert to using a simple representation
2017-09-06 09:01:53 -04:00
micael.bergeron
bca72f5906
wip: fake its a binary diff
2017-09-06 09:01:53 -04:00
Lin Jen-Shin
8e2350ae95
Raise encoding confidence threshold to 50
...
It is recommended that we set this to 50:
https://gitlab.com/gitlab-org/gitlab-ce/issues/35098#note_35036746
In this particular issue, the confidence was 42 for Shift JIS,
but in fact that's encoded in UTF-8 just with a single bad
character. In this case, we shouldn't try to treat it as Shift JIS,
but just treat it as UTF-8 and remove invalid bytes.
Treating it like Shift JIS would corrupt the whole data.
Unfortunately, the diff which would cause this could not be
disclosed therefore we can't use it as a test example.
2017-08-08 19:09:56 +08:00
James Lopez
6b53add3f9
Fix binary encoding error on MR diffs
2017-06-06 16:40:07 +00:00
Bob Van Landuyt
e564fe971f
Rename Gitlab::Git::EncodingHelper to Gitlab::EncodingHelper
2017-06-01 21:21:14 +00:00
