The previous "Content-Type" header parser was ported from Python and was
not very idiomatic Ruby. While fast and correct per the RFCs, it was
triggering bugs in Ruby MRI that we could probably work around. Ideally
someone would fix the bugs in Ruby MRI, but I haven't had time to track
them down.
Switch to using HTTP::Accept for parsing the media-type charset out of
the "Content-Type" header. Also relax the tests since HTTP::Accept is
somewhat stricter in the input it will accept. As a result, rest-client
will now ignore Content-Type headers with a trailing `;` character.
(This is invalid per the RFCs, so impact is likely to be small in
comparison to fixing the Ruby 2.4 memory leak.)
I also tried using https://github.com/httprb/content_type.rb but it is
significantly slower (caused 2x blowup in time on a simple benchmark)
and doesn't correctly handle content-types containing '.' characters.
Fixes: #523 (occasional MRI segfault)
Fixes: #611 (MRI 2.4.* memory leak)
* Unfortunately, HTTP::Accept does not support Ruby 2.0 due to its use
of named capture groups in StringScanner, which was added in Ruby 2.1.
Because rest-client still supports Ruby 2.0, fall back on the old
logic when running on Ruby 2.0. Even though Ruby 2.0 is unsupported,
it probably still sees wide use since it is the system Ruby even on
macOS Sierra.
* Don't bother running tests for .cgi_parse_header on Ruby 2.0. Still
keep the higher level tests that will exercise the deprecated code.
It turns out that Net::HTTP seems to always use UTF-8 as the default
encoding, even when Encoding.default_external is Encoding::IBM437 (as it
is on Windows/mingw).
Remove the old standalone implementation of Utils.encode_query_string in
favor of the unified version based on Utils.flatten_params.
This means that params in GET, POST x-www-form-urlencoded, and
POST multipart/form-data will all be handled in the same way (except for
escaping, since multipart/form-data is not escaped).
Begin the transition to unified HTTP parameter flattening by
reimplementing Payload::Base#flatten_params as Utils.flatten_params.
This new method uses a single recursive function rather than two
mutually recursive functions. It also establishes more unified handling
of URI escaping and nil or empty values.
Add a new method Utils.encode_query_string2 that is implemented in terms
of flatten_params. This will replace Utils.encode_query_string in the
next commit, but keep them both around in this commit for ease of
directly comparing their output to verify their equivalence.
Because of RestClient's use of Hashes to represent structured HTTP
parameters, there was previously no way to pass a structured ruby object
that contained the same key multiple times.
ParamsArray behaves similarly to Array, but enforces a little more
validation. It is intended to be used in place of a Hash in methods to
process HTTP parameters where duplicate keys are desired.
Switch the generation of HTTP GET params over to the new, more
featureful method in Utils, which handles Rack/Rails style nested
parameters. Also add a variety of tests for this functionality.
