External GCC for FreeBSD/mips

FreeBSD currently supports two compiler toolchains in the base system: Clang/LLVM of various versions (presently 6.0 in 12-CURRENT and 11-STABLE) and GCC 4.2.1 + binutils 2.17.50 (the last  versions released under GPLv2).  Clang/LLVM does not fully support all of FreeBSD's architectures, but the existing GNU toolchain is increasingly ancient.  It does not support C++11 or later standards, newer versions of DWARF, compressed debug symbols, ifuncs, etc.  Due to non-technical reasons, including a modern GNU toolchain in FreeBSD's base system is not feasible.  However, there has been ongoing work to support building a modern GNU toolchain from ports and installing it as the base system toolchain (e.g. /usr/bin/cc and /usr/bin/ld rather than /usr/local/bin/gcc<mumble>).

I've been working with this recently trying to polish up the existing bits and add a few missing pieces to make it viable using FreeBSD/mips (O32) as my initial testing platform since it runs fairly well under QEMU.

Building a sysroot

Cross building a toolchain via ports requires two things: an external toolchain to use on the build host and a system root ('sysroot') containing headers and libraries for the target host.  The external toolchain is available as a package from ports:

# pkg install mips-xtoolchain-gcc

The easiest way currently to build a sysroot is to build the world using the external toolchain and then install it:

# cd /usr/src

# make buildworld TARGET_ARCH=mips CROSS_TOOLCHAIN=mips-gcc WITHOUT_GCC=yes WITHOUT_BINUTILS=yes

# mkdir -p /ufs/mips/rootfs

# make installworld TARGET_ARCH=mips CROSS_TOOLCHAIN=mips-gcc WITHOUT_GCC=yes WITHOUT_BINUTILS=yes DESTDIR=/ufs/mips/rootfs

With these in place, one can now cross-build packages for a native toolchain.

Building Toolchain Packages

To build a native toolchain, three packages are required: ports-mgmt/pkg, base/binutils, and base/gcc.  Each package can be built via the following command:

# make CROSS_TOOLCHAIN=mips-gcc CROSS_SYSROOT=/ufs/mips/rootfs package

After the port finishes building the package, the package file can be found in the work/pkg subdirectory of the port.  Build all three packages and copy the packages someplace.  I put them in /ufs/mips/rootfs/root.

Using the Toolchain Packages

For my testing I fully populated the sysroot at /ufs/mips/rootfs (build and install MALTA kernel, 'make distribution', setup /etc/fstab and /etc/rc.conf) and then generated a disk image using makefs:

# makefs -M 32g -B be /ufs/mips/disk.img /ufs/mips/rootfs

I used this image as a disk for a QEMU instance:

# qemu-system-mips -kernel /ufs/mips/rootfs/boot/kernel/kernel \

    -nographic -drive file=/ufs/mips/disk.img,format=raw -m 2048 \

    -net nic -net user,hostfwd=tcp::8022-:22

Once QEMU is running, I can login to the virtual machine via ssh to port 8022 of the host.

To use the toolchain packages, you need to install them using pkg(8).  However, this requires manually bootstrapping pkg(8) itself.  It is probably simpler if you first disable the default pkg(8) repository following the instructions in /etc/pkg/FreeBSD.conf.  With that out of the way, manually extract the pkg-static binary and use it to install pkg:

# tar xf pkg-1.10.5.txz -C / /usr/local/sbin/pkg-static

# pkg-static install pkg-1.10.5.txz

Once pkg is installed, the binutils and gcc packages can be installed:

# pkg install freebsd-binutils-2.30_2.txz freebsd-gcc-6.3.0.txz

# cc --version

cc (GNU Collection for FreeBSD) 6.3.0

...

# ld --version

GNU ld (GNU Binutils) 2.30

...

Building a Native World

The next step I am still working on is testing building of newer worlds with the binutils and gcc packages installed.  Note that you have to be careful to ensure that you don't overwrite the packages you installed when eventually doing installworld, so to start with I added this to /etc/src.conf:

WITHOUT_GCC=yes

WITHOUT_BINUTILS=yes

I also wanted to force the build system to use /usr/bin/cc instead of trying to build GCC 4.2.1 and use it as the system compiler.  The current approach I'm using is to make a freebsd-gcc.mk cross toolchain make file in /usr/local/share/toolchains/freebsd-gcc.mk:

XCC=/usr/bin/cc

XCXX=/usr/bin/c++

XCPP=/usr/bin/cpp

CROSS_BINUTILS_PREFIX=/usr/bin/

X_COMPILER_TYPE=gcc

With this in place I tripped over one more bug which is that bsd.compiler.mk doesn't recognize the base/gcc compiler.  With that addressed I can now kick off a build world:

# make CROSS_TOOLCHAIN=freebsd-gcc buildworld TARGET_CPUTYPE=mips3

Unfortunately QEMU's timekeeping is a bit iffy and jemalloc keeps crashing due to an internal assertion failure which is slowing the build, but it does seem to make some progress on each run, so I keep restarting it with NO_CLEAN=yes.