This backports the https://github.com/rails/rails/pull/26815 and enables that for non-Rails5 and non-production environments.