Commit Graph

9 Commits

Author SHA1 Message Date
akr 7908180df1 * tool/transcode-tblgen.rb: record offsets array as index of
byte_array to avoid relocation.

* transcode.c (transcode_restartable0): add byte_array to get offsets
  array.

* transcode_data.h (BYTE_LOOKUP_BASE): change return type to
  uintptr_t.
  (rb_transcoder): add fields: byte_array, word_array and word_size.

* enc/trans/newline.trans: follow rb_transcoder change.

* enc/trans/iso2022.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19043 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01 17:40:32 +00:00
akr 752e053a1d * transcode_data.h (BYTE_LOOKUP): change to uintptr_t array.
(BYTE_LOOKUP_BASE): follow the type change.
  (BYTE_LOOKUP_INFO): ditto.
  (PType): ditto.
  (rb_transcoding): ditto.

* tool/transcode-tblgen.rb: follow the type change.

* transcode.c: ditto.

* enc/trans/newline.trans: ditto.

* enc/trans/iso2022.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19038 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-01 16:22:49 +00:00
akr 8841969485 * tool/transcode-tblgen.rb (transcode_generated_code): defined for
generating table at once.
  (transcode_tblgen): returns an empty string.
  (transcode_generate_node): ditto.

* enc/trans/newline.trans: use transcode_generated_code.

* enc/trans/iso2022.trans: ditto.

* enc/trans/single_byte.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.

* enc/trans/japanese.trans: ditto.

* enc/trans/korean.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19006 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-31 17:35:00 +00:00
akr c0d3881e0e * include/ruby/io.h (FMODE_TEXTMODE): defined.
* include/ruby/encoding.h (rb_econv_t): new field: flags.
  (rb_econv_binmode): declared.

* io.c (io_unread): text mode hack removed.
  (NEED_NEWLINE_DECODER): defined.
  (NEED_NEWLINE_ENCODER): defined.
  (NEED_READCONV): defined.
  (NEED_WRITECONV): defined.
  (TEXTMODE_NEWLINE_ENCODER): defined for windows.
  (make_writeconv): setup converter with TEXTMODE_NEWLINE_ENCODER for
  text mode.
  (io_fwrite): use NEED_WRITECONV.  character code conversion is
  disabled if fptr->writeconv_stateless is nil.
  (make_readconv): setup converter with
  ECONV_UNIVERSAL_NEWLINE_DECODER for text mode.
  (read_all): use NEED_READCONV.
  (appendline): use NEED_READCONV.
  (rb_io_getline_1): use NEED_READCONV.
  (io_getc): use NEED_READCONV.
  (rb_io_ungetc): use NEED_READCONV.
  (rb_io_binmode): OS-level text mode test removed.  call
  rb_econv_binmode.
  (rb_io_binmode_m): call rb_io_binmode_m with write_io as well.
  (rb_io_flags_mode): return mode string including "t".
  (rb_io_mode_flags): detect "t" for text mode.
  (rb_sysopen): always specify O_BINARY.

* transcode.c (rb_econv_open_by_transcoder_entries): initialize flags.
  (rb_econv_open): if source and destination encoding is
  both empty string, open newline converter.  last_tc will be NULL in
  this case.
  (rb_econv_encoding_to_insert_output): last_tc may be NULL now.
  (rb_econv_string): ditto.
  (output_replacement_character): ditto.
  (transcode_loop): ditto.
  (econv_init): ditto.
  (econv_inspect): ditto.
  (rb_econv_binmode): new function.




git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18780 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-22 16:44:00 +00:00
akr 7a0bea4fd3 * include/ruby/encoding.h (rb_econv_t): add fields: in_buf_start,
in_data_start, in_data_end, in_buf_end and last_trans_index.
  (rb_econv_output): removed.
  (rb_econv_insert_output): declared.
  (rb_econv_encoding_to_insert_output): declared.

* enc/trans/newline.trans (rb_universal_newline): stateful_type
  changed.

* transcode.c (transcode_restartable0): initialize inchar_start,
  tc->recognized_len and next_table at beginning of the loop.
  (rb_econv_open_by_transcoder_entries): initialize new fields.
  (rb_econv_open): setup last_trans_index.
  (trans_sweep): last out_buf_start can be non-NULL now.
  (rb_econv_convert): check last out_buf_start and in_buf_start at
  first.
  (rb_econv_output_with_destination_encoding): removed.
  (econv_just_convert): removed.
  (rb_econv_output): removed.
  (econv_primitive_output): method removed.
  (rb_econv_encoding_to_insert_output): new function.
  (allocate_converted_string): new function.
  (rb_econv_insert_output): new function.
  (econv_primitive_insert_output): new method.
  (output_replacement_character): use rb_econv_insert_output.  unused
  arguments removed.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18654 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-16 05:32:42 +00:00
akr a55e167b68 * transcode_data.h (rb_transcoder_stateful_type_t): defined.
(rb_transcoder): add field: stateful_type.

* tool/transcode-tblgen.rb: generate stateful_type field as
  stateless_converter.

* enc/trans/iso2022.trans: follow rb_transcoder change.

* enc/trans/newline.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18650 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-15 23:13:01 +00:00
akr 279e612973 fix variable name.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18558 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 05:53:20 +00:00
akr a14d5eb09b * enc/trans/newline.trans (rb_crlf_newline): new transcoder.
(rb_cr_newline): new transcoder.

* transcode.c (trans_open_i): one more exra room for input newline
  converter.
  (rb_trans_open): crlf newline and cr newline implemented.
  (Init_transcode): Encoding::Converter::CRLF_NEWLINE and
  Encoding::Converter::LF_NEWLINE defined.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18557 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 05:48:57 +00:00
akr 74a2a7bdbf * enc/trans/newline.trans: new file.
* transcode_data.h (rb_trans_t): add last_tc field.

* transcode.c (UNIVERSAL_NEWLINE): defined.
  (CRLF_NEWLINE): defined.
  (CR_NEWLINE): defined.
  (rb_trans_open_by_transcoder_entries): initialize last_tc.
  (trans_open_i): allocate one more room for newline converter.
  (rb_trans_open): universal newline implemented.
  (more_output_buffer): take max_output argument instead ts.
  (output_replacement_character): take tc argument instead of ts.
  (transcode_loop): use last_tc field.
  (econv_init): add flags argument for rb_trans_open.
  (Init_transcode): Encoding::Converter::UNIVERSAL_NEWLINE defined.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@18556 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-08-13 05:30:42 +00:00