* tool/transcode-tblgen.rb: 8bit byte of ASCII-8BIT is a valid
(but unique to ASCII-8BIT) character.
* transcode.c (rb_eConversionUndefined): new error.
(rb_eInvalidByteSequence): ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
resetting a state of stateful encoding.
* enc/trans/iso2022.trans (rb_EUC_JP_to_ISO_2022_JP): specify
finish_eucjp_to_iso2022jp for resetstate_func.
* tool/transcode-tblgen.rb: specify NULL for resetstate_func.
* transcode.c (output_replacement_character): call resetstate_func
before appending the replacement character.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18503 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
(rb_trans_elem_t): new type.
(rb_trans_t): new type.
* transcode.c (transcode_dispatch_cb): removed.
(transcode_dispatch): removed.
(rb_transcoding_result_t): moved to rb_trans_result_t in
transcode_data.h.
(transcode_restartable0): goto follow_info when FUNsi.
(rb_transcoding_open): use get_transcoder_entry.
(rb_trans_open): new function.
(rb_trans_conv): ditto.
(rb_trans_close): ditto.
(trans_open_i): ditto.
(trans_sweep): ditto.
(more_output_buffer): take rb_trans_t instead of rb_transcoding as
an argument.
(transcode_loop): take from_encoding and to_encoding instead of tr
as arguments. use rb_trans_open/rb_trans_conv/rb_trans_close.
(str_transcode): don't use transcode_dispatch.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* transcode.c (load_transcoder): extracted from transcode_dispatch_cb.
(rb_transcoding_result_t): renamed from transcode_result_t.
(rb_transcoding_open): new function.
(rb_transcoding_convert): ditto.
(rb_transcoding_close): ditto.
(transcode_loop): use rb_transcoding_open, rb_transcoding_convert
and rb_transcoding_close.
(str_transcode): don't need rb_transcoding.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18474 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* transcode.c (transcode_restartable0): renamed from
transcode_restartable.
save input buffer into feed buffer if next character is started the
point before input buffer. for example, "\x00\xd8\x01" then "\x02"
in UTF-16LE. \x02 causes invalid and next character is started from
\x01.
(transcode_restartable): new function to call
transcode_restartable0. if feed buffer is not empty, convert it at
first.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18467 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
as parameters.
(more_output_buffer): ditto.
(str_transcoding_resize): argument changed from rb_transcoding* to
VALUE.
(str_transcode): call transcode_loop with destination string and its
resize function.
* transcode_data.h (rb_transcoding): move ruby_string_dest and
flush_func to transcode_loop parameters.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18458 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
(rb_transcoder): preprocessor and postprocessor field removed.
change arguments of func_ii, func_si, func_io and func_so.
new field "finish_func".
* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
generatable.
* transcode.c (transcoder_lib_table): removed.
(transcoder_table): change structure.
(transcoder_key): removed because the above structure change.
(make_transcoder_entry): new function.
(get_transcoder_entry): ditto.
(rb_register_transcoder): follow the structure change.
(declare_transcoder): ditto.
(transcode_search_path): new function for breadth first search to
find a list of converters.
(transcode_search_path_i): new function.
(transcode_dispatch_cb): ditto.
(transcode_dispatch): use transcode_search_path.
(transcode_loop): follow the argument change.
(str_transcode): preprocessor and postprocessor stuff removed.
* enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion
re-implemented.
* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.
nute(23:52:53)% head -40 ChangeLog
Thu Aug 7 23:43:11 2008 Tanaka Akira <akr@fsij.org>
* transcode_data.h (rb_transcoding): new field "stateful".
(rb_transcoder): preprocessor and postprocessor field removed.
change arguments of func_ii, func_si, func_io and func_so.
new field "finish_func".
* tool/transcode-tblgen.rb: make FUNii, FUNsi and FUNio
generatable.
* transcode.c (transcoder_lib_table): removed.
(transcoder_table): change structure.
(transcoder_key): removed because the above structure change.
(make_transcoder_entry): new function.
(get_transcoder_entry): ditto.
(rb_register_transcoder): follow the structure change.
(declare_transcoder): ditto.
(transcode_search_path): new function for breadth first search to
find a list of converters.
(transcode_search_path_i): new function.
(transcode_dispatch_cb): ditto.
(transcode_dispatch): use transcode_search_path.
(transcode_loop): follow the argument change.
(str_transcode): preprocessor and postprocessor stuff removed.
* enc/trans/iso2022.erb.c: new file. ISO-2022-JP conversion
re-implemented.
* enc/trans/japanese.erb.c: ISO-2022-JP stuff removed.
* enc/trans/utf_16_32.erb.c: follow argument change of FUNso.
[ruby-dev:35798]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18419 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* transcode_data.h (rb_transcoding): include pointer to rb_transcoder
and auxiliary data.
* transcode_data.h (rb_transcoder): all callback functions shoud have
their own parameters.
* enc/trans/{japanese,single_byte}.c: constified.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15148 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* enc/trans/utf_16_32.c: new file, currently implementing
UTF-16BE conversions only.
* test/ruby/test_transcode.rb: Added tests for UTF-16BE;
made check_both_ways() use force_encoding differently.
* transcode_data.h, transcode.c: Support for more conversion
functions.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@15142 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* transcode.c (transcode_dispatch): reverted some of the changes
in r14746.
* transcode.c, enc/trans/single_byte.c: Added conversions to/from
US-ASCII and ASCII-8BIT (using data tables).
* enc/trans/single_byte.c: Some spacing/ordering changes due to
automatic data file generation.
* transcode_data.h, transcode.c: Preliminary code for using
micro-conversion functions.
* test/ruby/test_transcode.rb: Added some tests for US-ASCII and
ASCII-8BIT conversions.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14766 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* trancode.c: some minor formatting fixes
* transcode_data.h, transcode_data_iso_8859.c: Shortened
extremely frequently used macros to shorten file length.
* test/ruby/test_transcode.rb: Fixed name of test class;
added setup method to ensure all necessary encodings exist;
split tests into more test methods; added tests; fixed ordering
of arguments in assert_equal to have expected result first.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14236 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* transcode.c (transcoding): added a pointer to function to flush.
* transcode.c (transcode_loop): do not use string internal.
[ruby-dev:32512]
* transcode.c (str_transcode): allow Encoding objects.
* transcode_data.h (BYTE_LOOKUP): use actual struct name.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14176 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
