Fun with ARM64, Raspberry Pi and QEMU

Posted on .

I recently had a need to use a 64-bit ARM machine, running something like the Raspberry Pi OS. Not only that, I needed to run Docker on it to build a small base image of arm64v8/debian:bullseye-slim with the ca-certificates package installed.

The Docker build process in use is a multi-stage build, with a builder stage and a final runtime stage. The builder stage runs on amd64 and cross-compiles the codebase to arm64, then the runtime stage selects the arm64v8/debian:bullseye-slim image as the base, copies in the right artifacts and calls it a day.

When doing this, you have to be careful not to have any “RUN …” lines in the runtime stage as it’ll try to run the arm64 binaries on your amd64 machine, which is just asking for misery. So adding the ca-certificates to the final image is not simply sticking in a line like this:

RUN apt-get install -y ca-certificates

We want the equivalent of this, but without RUN’ning anything in the runtime stage.

There are two ways we could address this:

  1. Create an file, and have it call “apt-get install …” just before invoking the main binary

  2. Build a base image that already contains ca-certificates

Option 1 is okay when you don’t have arm64 machines lying around, but it’s not really satisfactory. Whenever you user starts a new container from this image, it’ll install ca-certificates and this will just be observed as slowdown. I really wanted option 2.

When you don’t have the hardware, what do you do? Emulate it! I found a cute little project called dockerpi dockerpi that gives you a Docker container running either a Pi 1, 2 or 3. Now the plan is this:

  1. Run the virtualized Pi
  2. Install Docker on it
  3. Build me a base image

The base image can be summed up like this:

FROM arm64v8/debian:bullseye-slim
RUN apt-get update -y
RUN apt-get install -y -qq --no-install-recommends ca-certificates

That’s it.

Everything went swimmingly, albeit slowly, until systemd failed to start the Docker service. The reason was something in the tune of “kernel does not have the right CONFIG_ flags”.

Now what? I started reading the dockerpi-entrypoint file for dockerpi, and started wondering if I couldn’t just perform those steps manually, but with a newer image of Raspberry Pi OS? I went and got the 2022-01-28-raspios-bullseye-arm64-lite.img, and started to piece together the missing bits.

We have a line like this where we need to fill in some placeholders:

qemu-system-aarch64 \
    -M raspi3b \
    -m 1G \
    -kernel <KERNEL> \
    -dtb <DTB> \
    --cpu arm1176 \
    -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootwait panic=1 dwc_otg.fiq_fsm_enable=0" \
    -netdev user,id=net0,hostfwd=tcp::5022-:22 -device usb-net,netdev=net0 \
    --drive "format=raw,file=2022-01-28-raspios-bullseye-arm64-lite.img" \
    --no-reboot \
    --display none \
    --serial mon:stdio

The first order of business is to resize the image to something larger. I went for 8GB.

$ qemu-img resize 2022-01-28-raspios-bullseye-arm64-lite.img 8GB

This should be a pretty quick operation. Next, we need to fish out a kernel and device tree overlay (dtb) from the image.

To do this, we need to realise that the .img file is basically just a binary blob that looks like a disk. We can run fdisk(8) on it!

$ fdisk -l 2022-01-28-raspios-bullseye-arm64-lite.img
Disk 2022-01-28-raspios-bullseye-arm64-lite.img: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xcb15ae4d

Device                                      Boot  Start      End  Sectors  Size Id Type
2022-01-28-raspios-bullseye-arm64-lite.img1        8192   532479   524288  256M  c W95 FAT32 (LBA)
2022-01-28-raspios-bullseye-arm64-lite.img2      532480 16777215 16244736  7.7G 83 Linux

The .img1 partition is what we want, that should be the boot sector containing both a kernel and device tree overlay. Let’s fish it out. The partition starts at 8192 (so we skip that many blocks), continues for 524288 blocks. The blocksize appears to be 512, so that’s what we’ll use:

$ dd if=2022-01-28-raspios-bullseye-arm64-lite.img of=fat.img bs=512 skip=8192 count=524288
524288+0 records in
524288+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 1.05331 s, 255 MB/s

A nice little sanity check here is to verify that it reports 256 MiB copied, same as the size of the partition reported by fdisk(8).

Now we want to open this up. I installed and used fatcat as the author of dockerpi did, but I suppose you could also set up the file as a loop-back device and simply mount it like any other drive. With fatcat you do:

$ fatcat -x fat/ fat.img

and it should spit out lots of lines with “Extracting …”. In the newly populated fat folder there should be a kernel8.img file and a file called bcm2710-rpi-3-b-plus.dtb. These are the files we need for our qemu-system-aarch64 line.

qemu-system-aarch64 \
    -M raspi3b \
    -m 1G \
    -kernel fat/kernel8.img \
    -dtb fat/bcm2710-rpi-3-b-plus.dtb \
    --cpu arm1176 \
    -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootwait panic=1 dwc_otg.fiq_fsm_enable=0" \
    -netdev user,id=net0,hostfwd=tcp::5022-:22 -device usb-net,netdev=net0 \
    --drive "format=raw,file=2022-01-28-raspios-bullseye-arm64-lite.img" \
    --no-reboot \
    --display none \
    --serial mon:stdio

and we should be good to go! If it launches then congratulations! You have a virtual Pi running. Log in with pi:raspberry and do a system update. Most important, remember to run raspi-config and (1) ask it to resize the disk to use all the available space and (2) start the SSH daemon so you can SSH to the VM.

To round out the Docker story, I managed to install Docker on it and it works! It’s hella slow, but that’s the virtualization world for you.


Articles from blogs I follow around the net

In praise of Plan 9

Plan 9 is an operating system designed by Bell Labs. It’s the OS they wrote after Unix, with the benefit of hindsight. It is the most interesting operating system that you’ve never heard of, and, in my opinion, the best operating system design to date. Even …

via Drew DeVault's blog November 12, 2022

Making Hare more debuggable

Hare programs need to be easier to debug. This blog post outlines our plans for improving the situation. For a start, we’d like to implement the following features: Detailed backtraces Address sanitization New memory allocator DWARF support These are rou…

via Blogs on The Hare programming language November 4, 2022

Go’s Version Control History

A tour of Go’s four version control systems.

via research!rsc February 14, 2022

Generated by openring