Commit graph

3 commits

Author SHA1 Message Date
Lin Jen-Shin
8e2350ae95 Raise encoding confidence threshold to 50
It is recommended that we set this to 50:
https://gitlab.com/gitlab-org/gitlab-ce/issues/35098#note_35036746

In this particular issue, the confidence was 42 for Shift JIS,
but in fact that's encoded in UTF-8 just with a single bad
character. In this case, we shouldn't try to treat it as Shift JIS,
but just treat it as UTF-8 and remove invalid bytes.

Treating it like Shift JIS would corrupt the whole data.

Unfortunately, the diff which would cause this could not be
disclosed therefore we can't use it as a test example.
2017-08-08 19:09:56 +08:00
James Lopez
6b53add3f9 Fix binary encoding error on MR diffs 2017-06-06 16:40:07 +00:00
Bob Van Landuyt
e564fe971f Rename Gitlab::Git::EncodingHelper to Gitlab::EncodingHelper 2017-06-01 21:21:14 +00:00