Running ARM Code on X86 Linux

Recently I was engaged in a project that penetrated a niche which we use to call “IoT” these days. Or, in simpler terms, there was some software running on low-power devices, collecting sensor data and sending JSONs to the cloud. I was responsible for packaging the device software. Cross-compiling was not an issue thanks to the great tooling on Debian (which was a target platform), but I had to be sure that the package installation, removal and upgrading works smoothly. Part of the task was to test that systemd-managed services start and stop as expected.

As you may have guessed, those low-power devices were not x86-based. They were the boxes running full Debian distro on ARM CPUs. Debian calls this port “armhf” (for “ARM hardfloat”, which simply means that when gcc compiles your code, the resulting binary will use a hardware floating point unit).

So my jenkins pipeline was building and packaging the software, but I had no idea how to make sure that this software will actually work as intended when delivered to the wild world outside. Unfortunately, access to the very ARM boxes was quite limited: we had one sample on-site, in use almost all the time by our embedded developer.

Of course, the solution was to use an emulator. I needed something that will execute those ARM binaries on my x86 laptop.

To illustrate the approach I took, let me first step back and show you an old quiz for system administrators.

Imagine someone accidentally removed all the execution bits from the chmod utility on your system. How would you make your /bin/chmod functional again?

Being a good quiz, it has several solutions. Let me demonstrate one of them – the most relevant to the topic of this article:

As you can see, there is a special kind of interpreter that can execute binaries.

In fact, both a script with a shebang and a compiled binary use the same kernel mechanics to run. Linux reads file’s “header” to determine its format and then calls a corresponding interpreter. There are several binary formats supported out of the box:

(ignis@ignis-pc|linux-4.19.2) find ./ -name 'binfmt*.c'
./arch/mips/kernel/binfmt_elfn32.c
./arch/mips/kernel/binfmt_elfo32.c
./arch/alpha/kernel/binfmt_loader.c
./fs/binfmt_em86.c
./fs/binfmt_elf_fdpic.c
./fs/binfmt_flat.c
./fs/binfmt_elf.c
./fs/binfmt_aout.c
./fs/binfmt_misc.c
./fs/binfmt_script.c

Notice binfmt_elf.c (ELF is a binary format used on linux) and binfmt_script.c.

But the most exciting thing here is the binfmt_misc. Think of it as of a plug-in system for binary formats.

With binfmt_misc it is possible to do

./DOOM.EXE

and have dosbox interpreting the exe-file.

In a similar fashion we can call a special interpreter for ARM binaries.

Qemu is a very powerful and feature-rich opensource software that emulates a ton of different CPU architectures using dynamic binary translation (this is what Java VM also does). Among those supported architectures is, of course, ARM as well. So in order to build a playground for ARM software on my x86 linux, I had to marry qemu with binfmt_misc magic. The good thing is: this is already a solved problem.

In my case (I use Manjaro distro) this is how I installed the neccessary packages:

yaourt -S qemu-arm-static binfmt-qemu-static

qemu-arm-static is, as the name suggests, a statically compiled build of the QEmu emulator for ARM. The other package simply configures binfmt_misc to use QEmu for ARM binaries.

All the funny binary formats supported by the binfmt_misc on your system can be listed like so:

(ignis@ignis-pc|~) ls -l /proc/sys/fs/binfmt_misc/
total 0
-rw-r--r-- 1 root root 0 30. Nov 14:58 aarch64
-rw-r--r-- 1 root root 0 30. Nov 14:58 appimage-type1
-rw-r--r-- 1 root root 0 30. Nov 14:58 appimage-type2
-rw-r--r-- 1 root root 0 30. Nov 14:58 arm
-rw-r--r-- 1 root root 0 30. Nov 14:58 armeb
-rw-r--r-- 1 root root 0 30. Nov 14:58 CLR
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-aarch64
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-alpha
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-arm
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-armeb
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-cris
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-m68k
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-microblaze
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-mips
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-mipsel
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-ppc
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-ppc64
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-ppc64abi32
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-s390x
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-sh4
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-sh4eb
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-sparc
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-sparc32plus
-rw-r--r-- 1 root root 0 30. Nov 14:58 qemu-sparc64
--w------- 1 root root 0 30. Nov 14:58 register
-rw-r--r-- 1 root root 0 30. Nov 14:58 status

The files here are the actual configuration interface for binfmt_misc.

(ignis@ignis-pc|~) cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/bin/qemu-arm-static
flags: 
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff
(ignis@ignis-pc|~) file /usr/bin/qemu-arm-static 
/usr/bin/qemu-arm-static: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, stripped

This setup is already enough to run a statically-linked ARM binary. It’s easy to prove with this tiny go program:

package main

import (
	"fmt"
	"runtime"
)

func main() {
	fmt.Printf("Hello from %s on %s!\n", runtime.GOARCH, runtime.GOOS)
}

Let’s cross-compile this piece of code and try to run it:

(ignis@ignis-pc|running-arm-code-on-x86) GOOS=linux GOARCH=arm go build hello_arm.go 
(ignis@ignis-pc|running-arm-code-on-x86) file hello_arm
arm: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, Go BuildID=EU8gFAuRaqyiFNHS24Mv/uyvsrKy_JSIa89DepWmG/fOI57lcKmfyalb7oGKtG/ynlSiaZtt4pSVD6q-QcO, with debug_info, not stripped
(ignis@ignis-pc|running-arm-code-on-x86) ./hello_arm 
Hello from arm on linux!

Heck, that was easy! But for a full experience (e.g. when we need shared libraries like most of the software does), we need a linux distro built for ARM.

You can use docker for that but in my case I needed more of a VM-experience to test how systemd services for the ARM software behave, so I opted for systemd-nspawn.

systemd-spawn is a true gem (and you don’t even have to install anything if your distro is systemd-based). systemd-spawn is extremely useful for local development and testing. Unlike docker, it doesn’t need any special images, it is more like chroot: we can just install debian in any directory and run from it.

In this example the directory in which we install debian distribution is called debian_jessie.

yaourt -S debootstrap
sudo debootstrap --foreign --arch=armhf jessie debian_jessie http://ftp.debian.org/debian/
sudo mount -t proc /proc debian_jessie/proc
sudo cp /usr/bin/qemu-arm-static debian_jessie/usr/bin/
sudo cp /etc/resolv.conf debian_jessie/etc
sudo PATH=/bin:/sbin:/usr/bin:/usr/sbin chroot debian_jessie /bin/bash

Once you’re in a chroot environment, finish the installation:

/debootstrap/debootstrap --second-stage

As this command concludes, it is time to set the root password:

sudo systemd-nspawn -D debian_jessie
passwd

At this point debian_jessie directory contains a fully workable debian system.

When I was testing new builds of our IoT software, I used to copy this initial installation into a separate directory to have a clean environment

sudo cp -r debian_jessie debian

and then dropped the files I needed inside

sudo cp ~/builds/*.deb ./debian/var/tmp

Starting an armhf port of debian jessie was then just a matter of

sudo systemd-nspawn -D debian /sbin/init

After a normal systemd boot process and a terminal login I had a fresh linux system ready for play.

It is just amazing how flexible linux as an ecosystem is and how much fun you can have by digging deeper in it. I hope you had fun too reading this article. Enjoy your day!

For some more interesting use cases for systemd-nspawn, check out this article