[rubygems/rubygems] Mask the file mode when extracting files

When extracting files from the tarball, a mode is retrieved from
the header. Occasionally you'll encounter a gem that was packaged
on a system whose permission bits result in a value that is larger
than the value that File.chmod will allow (anything >= 2^16). In
that case the extraction fails with a RangeError, which is pretty
esoteric.

If you extract the tarball with the tar and gunzip utilities, the
file permissions end up being just the bottom 16 bits masked off
from the original value. I've mirrored that behavior here. Per the
tar spec:

> Modes which are not supported by the operating system restoring
> files from the archive will be ignored.

I think that basically means what I've done here.

---

This commit also changes the behavior very slightly with regard to
when the chmod is called. Previously it was called while the file
descriptor was still open, but after the write call.

When write flushes, the file permissions are changed to the mode
value from the File.open call, undoing the changes made by
FileUtils.chmod. CRuby appears to flush the buffer after the
chmod call, whereas TruffleRuby flushes before the chmod call.
So the file permissions can change depending on implementation.
Both implementations end up getting the correct file permissions
for the bottom 9 bits (user, group, world), but differ with
regard to the sticky bit in the next 3.

To get consistent behavior, this commit changes it to close the
file descriptor before attempting to chmod anything, which makes
it consistent because the write flushes in both cases.

https://github.com/rubygems/rubygems/commit/22ce076e99

lib/rubygems/package.rb

@@ -444,10 +444,10 @@ EOM
           directories << mkdir
         end
 
-        File.open destination, "wb" do |out|
-          out.write entry.read
+        if entry.file?
+          File.open(destination, "wb") {|out| out.write entry.read }
           FileUtils.chmod file_mode(entry.header.mode), destination
-        end if entry.file?
+        end
 
         verbose destination
       end
@@ -467,7 +467,12 @@ EOM
   end
 
   def file_mode(mode) # :nodoc:
-    ((mode & 0111).zero? ? data_mode : prog_mode) || mode
+    ((mode & 0111).zero? ? data_mode : prog_mode) ||
+      # If we're not using one of the default modes, then we're going to fall
+      # back to the mode from the tarball. In this case we need to mask it down
+      # to fit into 2^16 bits (the maximum value for a mode in CRuby since it
+      # gets put into an unsigned short).
+      (mode & ((1 << 16) - 1))
   end
 
   ##

test/rubygems/test_gem_package.rb

@@ -510,6 +510,21 @@ class TestGemPackage < Gem::Package::TarTestCase
     assert_path_exist @destination
   end
 
+  def test_extract_file_permissions
+    pend "chmod not supported" if win_platform?
+
+    gem_with_long_permissions = File.expand_path("packages/Bluebie-legs-0.6.2.gem", __dir__)
+
+    package = Gem::Package.new gem_with_long_permissions
+
+    package.extract_files @destination
+
+    filepath = File.join @destination, "README.rdoc"
+    assert_path_exist filepath
+
+    assert_equal 0104444, File.stat(filepath).mode
+  end
+
   def test_extract_tar_gz_absolute
     package = Gem::Package.new @gem