Fun with ARM64, Raspberry Pi and QEMU

Posted on 2022-02-23

I recently had a need to use a 64-bit ARM machine, running something like the Raspberry Pi OS. Not only that, I needed to run Docker on it to build a small base image of arm64v8/debian:bullseye-slim with the ca-certificates package installed.

The Docker build process in use is a multi-stage build, with a builder stage and a final runtime stage. The builder stage runs on amd64 and cross-compiles the codebase to arm64, then the runtime stage selects the arm64v8/debian:bullseye-slim image as the base, copies in the right artifacts and calls it a day.

When doing this, you have to be careful not to have any “RUN …” lines in the runtime stage as it’ll try to run the arm64 binaries on your amd64 machine, which is just asking for misery. So adding the ca-certificates to the final image is not simply sticking in a line like this:

RUN apt-get install -y ca-certificates

We want the equivalent of this, but without RUN’ning anything in the runtime stage.

There are two ways we could address this:

Create an entrypoint.sh file, and have it call “apt-get install …” just before invoking the main binary
Build a base image that already contains ca-certificates

Option 1 is okay when you don’t have arm64 machines lying around, but it’s not really satisfactory. Whenever you user starts a new container from this image, it’ll install ca-certificates and this will just be observed as slowdown. I really wanted option 2.

When you don’t have the hardware, what do you do? Emulate it! I found a cute little project called dockerpihttps://github.com/lukechilds/dockerpi that gives you a Docker container running either a Pi 1, 2 or 3. Now the plan is this:

Run the virtualized Pi
Install Docker on it
Build me a base image

The base image can be summed up like this:

FROM arm64v8/debian:bullseye-slim
RUN apt-get update -y
RUN apt-get install -y -qq --no-install-recommends ca-certificates

That’s it.

Everything went swimmingly, albeit slowly, until systemd failed to start the Docker service. The reason was something in the tune of “kernel does not have the right CONFIG_ flags”.

Now what? I started reading the entrypoint.sh dockerpi-entrypoint file for dockerpi, and started wondering if I couldn’t just perform those steps manually, but with a newer image of Raspberry Pi OS? I went and got the 2022-01-28-raspios-bullseye-arm64-lite.img, and started to piece together the missing bits.

We have a line like this where we need to fill in some placeholders:

qemu-system-aarch64 \
    -M raspi3b \
    -m 1G \
    -kernel <KERNEL> \
    -dtb <DTB> \
    --cpu arm1176 \
    -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootwait panic=1 dwc_otg.fiq_fsm_enable=0" \
    -netdev user,id=net0,hostfwd=tcp::5022-:22 -device usb-net,netdev=net0 \
    --drive "format=raw,file=2022-01-28-raspios-bullseye-arm64-lite.img" \
    --no-reboot \
    --display none \
    --serial mon:stdio

The first order of business is to resize the image to something larger. I went for 8GB.

$ qemu-img resize 2022-01-28-raspios-bullseye-arm64-lite.img 8GB

This should be a pretty quick operation. Next, we need to fish out a kernel and device tree overlay (dtb) from the image.

To do this, we need to realise that the .img file is basically just a binary blob that looks like a disk. We can run fdisk(8) on it!

$ fdisk -l 2022-01-28-raspios-bullseye-arm64-lite.img
Disk 2022-01-28-raspios-bullseye-arm64-lite.img: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcb15ae4d

Device                                      Boot  Start      End  Sectors  Size Id Type
2022-01-28-raspios-bullseye-arm64-lite.img1        8192   532479   524288  256M  c W95 FAT32 (LBA)
2022-01-28-raspios-bullseye-arm64-lite.img2      532480 16777215 16244736  7.7G 83 Linux

The .img1 partition is what we want, that should be the boot sector containing both a kernel and device tree overlay. Let’s fish it out. The partition starts at 8192 (so we skip that many blocks), continues for 524288 blocks. The blocksize appears to be 512, so that’s what we’ll use:

$ dd if=2022-01-28-raspios-bullseye-arm64-lite.img of=fat.img bs=512 skip=8192 count=524288
524288+0 records in
524288+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 1.05331 s, 255 MB/s

A nice little sanity check here is to verify that it reports 256 MiB copied, same as the size of the partition reported by fdisk(8).

Now we want to open this up. I installed and used fatcat⊕fatcat is a forensic tool to explore, extract, and repair FAT filesystems as the author of dockerpi did, but I suppose you could also set up the file as a loop-back device and simply mount it like any other drive. With fatcat you do:

$ fatcat -x fat/ fat.img

and it should spit out lots of lines with “Extracting …”. In the newly populated fat folder there should be a kernel8.img file and a file called bcm2710-rpi-3-b-plus.dtb. These are the files we need for our qemu-system-aarch64 line.

qemu-system-aarch64 \
    -M raspi3b \
    -m 1G \
    -kernel fat/kernel8.img \
    -dtb fat/bcm2710-rpi-3-b-plus.dtb \
    --cpu arm1176 \
    -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootwait panic=1 dwc_otg.fiq_fsm_enable=0" \
    -netdev user,id=net0,hostfwd=tcp::5022-:22 -device usb-net,netdev=net0 \
    --drive "format=raw,file=2022-01-28-raspios-bullseye-arm64-lite.img" \
    --no-reboot \
    --display none \
    --serial mon:stdio

and we should be good to go! If it launches then congratulations! You have a virtual Pi running. Log in with pi:raspberry and do a system update. Most important, remember to run raspi-config and (1) ask it to resize the disk to use all the available space and (2) start the SSH daemon so you can SSH to the VM.

To round out the Docker story, I managed to install Docker on it and it works! It’s hella slow, but that’s the virtualization world for you.