sortix--sortix/doc/cross-compilation-sins

Cross Compilation Sins
======================

Cross-compilation is the act of compiling a program for execution on another
machine that is different from the current machine. The other machine can have
another processor, another operating system, or just another directory structure
with different programs and libraries installed. The important part is that the
cross-compiled program is not meant to be executed on the build machine and
cannot be assumed to execute correctly on the build machine.

Cross-compilation works in concept simply by substituting the usual compiler
with a special cross-compiler targeting the host machine rather than the build
machine. The development headers and libraries would then already be installed
in the system root used by the cross-compiler. The program will then be built
normally and the result is a binary for the host machine.

Unfortunately, a lot of software attempt to be too clever for their own good and
rely on assumptions that are not valid in the face of cross-compilation. This is
a list of such sins that software packages occasionally make that prevent proper
cross-compilation. Almost all software ought to cross-compile cleanly.

Not supporting cross-compilation
--------------------------------

Programs should use a build system that allows cross-compiler, or be stupid and
just use the specified compiler without violating rules in this document. Build
systems should not hard-code a particular compiler, but use a CC or CXX
environmental variable that possibly default to reasonable values, or using the
full ./configure machinery for detecting the correct compiler.

Root is the empty prefix
------------------------

The build system must treat the empty prefix as distinct from the default prefix
(which is usually /usr/local). A prefix is a string of characters that is added
before the usual installation directories such as /bin and /lib. To install a
program into the /bin directory (in the root filesystem), you need add no
characters in front of /lib. Thus, the empty prefix is the root directory. The
root directory is the prefix of /, as that would install the program into //bin.
Indeed //bin is not only ugly, but it actually has a special alternate meaning
than the root directory on some operating systems.

Probing root filesystem
-----------------------

The actual root filesystem of the host machine is not available at build time.
That means that probing system files are out of the question and should be
recoded into a runtime check at program startup. For instance, checking special
directories such as /dev or /proc, or system configuration in /etc, even
manually locating headers in /usr/include and libraries in /usr/lib is not
allowed (when though they would be there in the system root, potentially).

Instead build systems should view things through the eyes of the cross-compiler
that may use exotic strategies to locate the desired resources (perhaps the
cross-compiler is an executable that wraps the real compiler and adds special
-I and --sysroot options, for instance). To check whether a header is available,
the build system should preprocess a file that #includes it. To check whether a
library is available, the build system should try to link an executable against
it.

This discussion even covers looking at the configured prefix and other
installation directories, even if prefixed with $DESTDIR, except for the purpose
of installing files there.

Executing cross-compiled programs
---------------------------------

It is not possible to run a cross-compiled program when cross-compiling. The
results vary, maybe the program works, maybe the program loads but does
something unexpected, maybe the program loads and enter an infinite loop, maybe
the program loads and crashes, or maybe the program doesn't even load. The
important part is that the build system should not even attempt to execute any
programs it compiles. Some packages use special configure-time tests to compile
test programs to check the behavior of the host machine, such as whether a
standard function works.

Such tests are fundamentally broken if they require executing the test program;
usually it is sufficient to just check if the program links. This means that
tests about the run-time behavior of programs are not possible on the build
machine. If possible, such tests could be delayed until program startup.

However, such tests are usually to detect broken systems. If testing is not
possible, then the build system should assume that the system is not broken. It
is often just a single operating system that has such a problem and it may be
fixed in later releases. It is acceptable to inconvenience users of broken
operarating systems by asking them to pass an --using-broken-os configure option
or something similar, as long as it doesn't inconvenience honest users of good
operating systems.

$DESTDIR
--------

Programs must support the DESTDIR environmental variable at package installation
time. It is an additional temporary prefix that is added to all the installation
directory paths when installing files and directories. It is commonly used for
package management that wishes to stage programs at a temporary location rather
than installing into the final destination immediately. It is also used for
cross-compilation where installation into the local machine's root directory
would be disastrous, where you rather want to use the cross-compilation system
root as DESTDIR.

It is important to understand that the prefix set at configure time is the
location where the program will end up on the host machine, not the installation
directory on the build machine. The installation directory on the build machine
would be the prefix given at configure time prefixed with DESTDIR being the
system root.

If packages do not support DESTDIR, it is possible to work-around the issue by
adding the intended DESTDIR to the prefix given at configure time. This works
only as long as the program doesn't remember the prefix or store it in system
configuration files. The better solution is just to patch in $(DESTDIR) support.

Cross-compiled Build Tools
--------------------------

Programs often need special-purpose build tools to compile some aspects of them
or associated files or documentation. If such a program is local to the project
and not a stand-alone tool on its own right, then it is not uncommon for the
build system to build first the tool and then use the tool during the build of
the real program.

This presents a crucial problem for cross-compilation: Two compilers must be in
play. Otherwise, the build system may helpfully use the cross-compiler to
cross-compile the build tool and attempt to execute the cross-compiled build
tool (which is intended to run on the host machine, not the build machine). This
is a common problem that prevents otherwise cross-compilable programs from
being cross-compilable.

The solution is to detect and use two compilers: The compiler for the build
machine and the compiler for the host machine. This introduces some new
complexity, but autoconf generated ./configure scripts can deal rather easily
with it. There is a problem, though, if the build tool itself has dependencies
on external projects besides the standard library, as that would mean the build
system would need to detect and handle dependencies for both the build and host
machine. I'm not even sure the autoconf configure style --with-foo options are
fine-grained enough to support --with-build-foo and with-host-foo options if
they differ.

A better solution is perhaps to promote the custom build tool to a general
purpose or at least special purpose tool that is installed along with the
program, while allowing the special purpose tool to be built separately and
just that tool. This allows the user wishing to cross-compile to first build the
custom build tool locally on his configure, install it on the build machine, and
then have the tool in $PATH during the actual cross-compilation of the program.
This way the build system doesn't need to have two compilers in play, but at the
cost of essentially splitting the project into two subprojects: The real program
and the custom build tool. This option is preferable if the custom build tool
can be adapted so it is reusable by other projects.

You can also change the implementation language of the build tool to an
interpreted language, such as a shell script, python or anything suitable. It
would be prudent to ensure such interpreted languages can also be cross-compiled
cleanly.

Degrading functionality of program
----------------------------------

Programs should not be partially cross-compilable, with optional parts of the
program not available if cross-compiled. In the event that such optional parts
cannot be cross-compiled, it might be because they are violating rules in this
document or an external dependency does.

Degrading quality of program
----------------------------

Programs that are cross-compiled should be as similar as possible to the case
where they are not cross-compiled. However, it is acceptable if there is a
performance loss if the program needs to do run-time checking when a test is not
possible at compile time or other cases where the build system needs to make a
decision and insufficient data is available and both solutions would work
correctly.

Custom Configure Scripts
------------------------

Some perhaps ship with a custom hand-written ./configure script as opposed to a
script generated by tools such as autoconf. It doesn't matter in what language
a configure script is written (as long as it can be correctly executed) or
whether it is generated or hand-written. However, it is important that it
correctly implement the standard semantics customary with GNU autoconf generated
./configure scripts. In particular, for this discussion, it must support the
--build, --host and --target options, as well as all the standard installation
directory options (--prefix, --exec-prefix, ...). It is also important that it
correctly locate a cross-compiler through the --host option by using it as a
cross-tool prefix. For instance, --host=x86_64-sortix must correctly locate
the x86_64-sortix-gcc compiler.

Remembering the Compiler
------------------------

Unusually, some libraries remember the compiler executable and used compiler
options and store them in special foo-config files or even installed system
headers. This must never be done as the cross-compiler is a temporary tool. The
library is meant to be used on the host system, it would be odd if programs
depending on the library attempted to use a cross-compiler when building such
programs on the host machine. It also violates the principle that which compiler
is used is decided by the user, rather than secretly by the package.

Making cross-compilation needlessly hard
----------------------------------------

Some build systems require the user to answer particular questions about the
host machine, while is perfectly capable of automatically answering such
questions about the build machine. This occasionally takes the form of autoconf
generated ./configure scripts requiring the user to set autoconf cache values
that answer runtime tests. The obvious solution is to do runtime tests at
program startup instead or more careful tests that are possible at compile time.

Other problems include all sorts of miscellaneous situations where the user is
required to jump through hoops to cross-compile a program, when it could have
been much simpler if the build system had followed standard patterns. This is
often a symptom of projects where cross-compilation is considered unusual and
special-purpose, rather than than a natural state of things if you don't assume
particular runtime tests are possible at compile time.

pkg-config
----------

Libraries should install pkg-config files rather than libtool .la files or
foo-config scripts as neither of those approaches support cross-compilation or
system roots, while pkg-config is perfectly aware of such use cases.

Programs should never look for libtool .la files or use foo-config scripts for
the same reasons. It is too possible that the program ends up finding a tool for
the build machine instead, if the installed foo-config script wasn't in the
user's PATH. Fortunately, the invocation of foo-config scripts and pkg-config
are usually similar enough, so it is simple to adapt a build system to use the
pkg-config variant exclusively instead.

The user can build a special cross-pkg-config or wrap an existing pkg-config by
setting special environmental variables. There are some ceveats if the program
builds custom build tools that needs dependencies detected through pkg-config.
In that case, the user may need to have a special pkg-config with a tool prefix
or pass a configure option setting the name of the build machine pkg-config
script.

libtool .la files
-----------------

Libraries managed with libtool often install special .la files into the
configured libdir. These files contain information on how to link against the
library and what compiler options to use. Unfortunately they don't support
cross-compilation and system roots. As such, too easily the compile process will
begin attempting to link against files relative to the root directory on the
build machine.

The recommendation is to kill such files on sight and to never generate them in
the first place, and certainly to never install them into the system root. It is
usually safe to delete such files, especially if the library installs pkg-config
files or if you install into well-known system directories.

foo-config
----------

Libraries occasionally install special foo-config scripts into the configured
bindir. These files are executable shell scripts that output the compiler
options that should be used to link against the library. Unfortunately they
don't support cross-compilation and system roots. As such, too easily the
compile process will begin attempting to link against files relative to the root
directory on the build machine.

The recommendation is to kill such files on sight and to never generate them in
the first place, and certainly to never install them into the system root. It is
usually safe to delete such files, especially if the library installs pkg-config
files or if you install into well-known system directories. Watch out for
programs linking against the library that wrongly locate foo-config scripts in
your $PATH (which potentially come from your distribution). Such programs needs
to be patched to use pkg-config instead.