Cross compiling Rust with Drone

Published on 30 December 2022 - ci development rust technology

Why

For a while now, I have been working on a personal project called newsletter2web, which I run on a Raspberry PI. As the title already indicates, it is written in Rust.

Since compiling the project takes a while, especially on a Raspberry PI, I decided to investigate how to make it build automatically.

That sounds like a job for continuous integration solution. But these tend to use quite a bit of resources, even when not in use. That is wasteful and I didn't want that.

So put together, I got the following requirements:

A continuous integration solution, that
Would use little resources when not in use,
Integrates with Gitea, and
Is capable of cross-compiling Rust

After a bit of digging, I found Drone CI to be a nice fit: it is fairly lightweight, and easy to get running. Making it work well enough to cross compile Rust took a while though.

This post describes the cross compilation, the Drone and Gitea setup is described in the previous post.

Definitions

Since not everyone may be familiar with cross compilation, it is useful to know some definitions.

Host: the machine in which cross compilation is performed.
Target: the operating system and CPU architecture for which a project is compiled.
Toolchain: the set of software required for compilation. This is at least a linker, but usually also a C compiler and target-specific source code are required.

When cross compiling, this is usually a different set of software than is required to compile for the host itself.

How does it work

Rust has good support for cross compilation already out of the box. However, in order to use it, the compilation toolchains need to be installed on the host. Also, the Rust project sometimes needs custom configuration as well, in order to be able to use this.

It can be tedious to do this manually. Therefore, we use cross, which is a project that uses pre-built Docker images for each target. Cross still needs to have the Rust toolchain for each target installed though.

Since cross uses Docker, and compiling source code has the possibility of executing arbitrary code on the host, a security boundary is required as well. For that, a Docker in Docker setup is used.

Docker in docker means running the Docker service itself inside a Docker container running on the host.

Since cross needs the Rust toolchain for each target, we will use a custom Docker image to run it.

The Docker image

This Docker image will have everything installed that is required to run cross. So it contains the Rust toolchain for each target and the Docker service for Docker in Docker.

This is the corresponding Dockerfile:

# Base image. Change this to use another Rust version.
FROM rust:1.66-slim

# Install Docker (for docker-in-docker)
RUN apt-get update \
 && apt-get install -y ca-certificates curl gnupg \
 && mkdir -p /etc/apt/keyrings \
 && curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
 && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian bullseye stable" >> /etc/apt/sources.list.d/docker.list \
 && apt-get update \
 && apt-get install -y docker-ce-cli \
 && apt-get clean

# Install cross v0.2.4
RUN mkdir -p /usr/local/cargo/bin && curl -L https://github.com/cross-rs/cross/releases/download/v0.2.4/cross-x86_64-unknown-linux-gnu.tar.gz | tar -xz -C /usr/local/cargo/bin

# (Optional) Pre-install rust targets
# This significantly reduces build time, at the cost of an increased image size
RUN rustup target add \
        aarch64-unknown-linux-gnu \
        aarch64-unknown-linux-musl \
        x86_64-pc-windows-gnu \
        x86_64-unknown-linux-gnu \
        x86_64-unknown-linux-musl \
    && rustup component add --target aarch64-unknown-linux-gnu rust-src \
    && rustup component add --target aarch64-unknown-linux-musl rust-src \
    && rustup component add --target x86_64-pc-windows-gnu rust-src \
    && rustup component add --target x86_64-unknown-linux-gnu rust-src \
    && rustup component add --target x86_64-unknown-linux-musl rust-src

Don't forget to push it to a registry, so it can be used by Drone later.

The Drone build pipeline

The build pipeline is where the real magic happens. I will use a .drone.yml file here. In production I use a Jsonnet file to generate this. You can find that in the newsletter-to-web repository.

The pipeline is long, so it can be found at the end of this post.

Docker in Docker service

Before starting the builds, a running Docker agent is required. This is handled by starting Docker in Docker as a service, and then waiting until it is ready by a separate build step. All the cross compilation build steps then wait for that to happen.

First, a shared volume is created for the Docker socket. That way, the agent and all build steps can find each other.

# Shared volume for the Docker in Docker service and build steps.
# Without this, cross won't be able to find a running Docker agent.
volumes:
- name: dockersock
  temp: {}

Then, the Docker in Docker service is defined. It uses the volume as described. It will pull the Docker in Docker image and run it.

This requires privileged mode, because otherwise pulling images and creating volumes fails.

services:
- image: docker:dind
  name: docker
  privileged: true   # Docker in Docker requires this, sadly.
  pull: true
  volumes:
  - name: dockersock
    path: /var/run

Then, in the steps, the Wait for Docker step is defined. This will try to run docker image ls until it succeeds. Once it is, it will print information about the current Docker installation, and pull an image. This is essentially a healt check.

# Ensure Docker agent has started before continuing.
steps:
- name: Wait for Docker
  image: rust-dind-cross:1.66-full
  pull: true
  commands:
  - mkdir artifacts
  - while ! docker image ls; do sleep 1; done
  - docker info
  - docker pull hello-world:latest
  volumes:
  - name: dockersock
    path: /var/run

All other build steps can then wait until it is ready by depending on this build step. Like so:

steps:
- name: Build for amd64
  depends_on:
  - Wait for Docker

Cross compilation

A cross compilation step is defined like so (under the steps definition):

name: Build for windows-amd64
image: rust-dind-cross:1.66-full
pull: true
environment:
  CROSS_REMOTE: true
depends_on:
- Wait for Docker
volumes:
- name: dockersock
  path: /var/run
commands:
- cross build --release --target x86_64-pc-windows-gnu
- cp target/x86_64-pc-windows-gnu/release/drone-test.exe artifacts/drone-test-windows-amd64.exe
- rm -rf target/x86_64-pc-windows-gnu/release/*

It starts with defining the docker image to use image: rust-dind-cross:1.66-full. This is the images that was created before the section The Docker image.

Then, it sets the environment:

environment:
  CROSS_REMOTE: true

This tells cross that it is running in a remote docker, so it will copy everything necessary for the build into the build image that it creates.

As described before, it depends on the Docker in Docker agent, so that is defined as well, together with the shared volume:

depends_on:
- Wait for Docker
volumes:
- name: dockersock
  path: /var/run

Finally, the commands to run are listed:

commands:
- cross build --release --target x86_64-pc-windows-gnu
- cp target/x86_64-pc-windows-gnu/release/drone-test.exe artifacts/drone-test-windows-amd64.exe
- rm -rf target/x86_64-pc-windows-gnu/release/*

This will tell cross to build a release binary for the given target. In order to do so, it will start its own Docker container, copies all the source files into it, and then runs a normal Rust build.

The container started by cross is the container that contains the required build toolchain for the specific target, as described in the definitions.

When it is finished compiling, the binary is copied to an artifacts/ directory for later reference. It will now also gain a postfix with the target it was build for. Otherwise, all binaries would be named the same and would be overwritten.

Finally, the release directory is cleaned up in order to save disk space.

When all cross compilations have succeeded, the built artifacts are shown. This step is not necessary, and you could do anything else with it.

name: Show built artifacts
image: rust-dind-cross:1.66-full
commands:
- ls -lah artifacts
depends_on:
- Build for arm64-gnu
- Build for arm64-musl
- Build for windows-amd64
- Build for amd64-gnu
- Build for amd64-musl

The interesting part here, is that it depends on all the cross compilation build steps.

When tagged, create a Gitea release

When the pipeline is run for a tag, than as the very last step a new draft release is created in Gitea. This can then be edited and published.

name: Create release on gitea
image: plugins/gitea-release
settings:
  api_key:
    from_secret: gitea_token
  base_url: https://gitea.example.com
  checksum: sha256
  files: artifacts/*
  draft: true
depends_on:
- Build for arm64-gnu
- Build for arm64-musl
- Build for windows-amd64
- Build for amd64-gnu
- Build for amd64-musl
when:
  event:
  - tag

This steps depends on a successful build of all cross compilation steps. It uses a drone plugin called Gitea Release.

It will:

Upload all files in the artifacts/ directory as assets.
Calculate the SHA256 sum of all artifacts, and add that as a sha256sum.txt file to the assets.

This only happens when the pipeline runs for a git tag, which is described like this:

when:
  event:
  - tag

Full build pipeline

Contents of .drone.yml

kind: pipeline
type: docker
name: 'Rust cross compilation'
platform:
  arch: amd64

image_pull_secrets:
- docker_private_repo

# Shared volume for the Docker in Docker service and build steps.
# Without this, cross won't be able to find a running Docker agent.
volumes:
- name: dockersock
  temp: {}

services:
- image: docker:dind
  name: docker
  privileged: true   # Docker in Docker requires this, sadly.
  pull: true
  volumes:
  - name: dockersock
    path: /var/run

# Ensure Docker agent has started before continuing.
steps:
- name: Wait for Docker
  image: rust-dind-cross:1.66-full
  pull: true
  commands:
  - mkdir artifacts
  - while ! docker image ls; do sleep 1; done
  - docker info
  - docker pull hello-world:latest
  volumes:
  - name: dockersock
    path: /var/run

# Build for aarch64-unknown-linux-gnu (arm64)
- name: Build for arm64-gnu
  image: rust-dind-cross:1.66-full
  pull: true
  environment:
    CROSS_REMOTE: true
  commands:
  - cross build --release --target aarch64-unknown-linux-gnu
  - cp target/aarch64-unknown-linux-gnu/release/drone-test artifacts/drone-test-arm64-gnu
  - rm -rf target/aarch64-unknown-linux-gnu/release/*
  depends_on:
  - Wait for Docker
  volumes:
  - name: dockersock
    path: /var/run

# Build for aarch64-unknown-linux-musl (arm64 with Alpine linux)
- name: Build for arm64-musl
  image: rust-dind-cross:1.66-full
  pull: true
  environment:
    CROSS_REMOTE: true
  commands:
  - cross build --release --target aarch64-unknown-linux-musl
  - cp target/aarch64-unknown-linux-musl/release/drone-test artifacts/drone-test-arm64-musl
  - rm -rf target/aarch64-unknown-linux-musl/release/*
  depends_on:
  - Wait for Docker
  volumes:
  - name: dockersock
    path: /var/run

# Build for x86_64-pc-windows-gnu
# This generates a binary that can run on Windows.
-name: Build for windows-amd64
  image: rust-dind-cross:1.66-full
  pull: true
  environment:
    CROSS_REMOTE: true
  commands:
  - cross build --release --target x86_64-pc-windows-gnu
  - cp target/x86_64-pc-windows-gnu/release/drone-test.exe artifacts/drone-test-windows-amd64.exe
  - rm -rf target/x86_64-pc-windows-gnu/release/*
  depends_on:
  - Wait for Docker
  volumes:
  - name: dockersock
    path: /var/run

# Build for x86_64-unknown-linux-gnu (amd64)
- name: Build for amd64-gnu
  image: rust-dind-cross:1.66-full
  pull: true
  environment:
    CROSS_REMOTE: true
  commands:
  - cross build --release --target x86_64-unknown-linux-gnu
  - cp target/x86_64-unknown-linux-gnu/release/drone-test artifacts/drone-test-amd64-gnu
  - rm -rf target/x86_64-unknown-linux-gnu/release/*
  depends_on:
  - Wait for Docker
  volumes:
  - name: dockersock
    path: /var/run

# Build for x86_64-unknown-linux-musl (amd64 with Alpine linux)
- name: Build for amd64-musl
  image: rust-dind-cross:1.66-full
  pull: true
  environment:
    CROSS_REMOTE: true
  commands:
  - cross build --release --target x86_64-unknown-linux-musl
  - cp target/x86_64-unknown-linux-musl/release/drone-test artifacts/drone-test-amd64-musl
  - rm -rf target/x86_64-unknown-linux-musl/release/*
  depends_on:
  - Wait for Docker
  volumes:
  - name: dockersock
    path: /var/run

# Show all built artifacts
- name: Show built artifacts
  image: rust-dind-cross:1.66-full
  commands:
  - ls -lah artifacts
  depends_on:
  - Build for arm64-gnu
  - Build for arm64-musl
  - Build for windows-amd64
  - Build for amd64-gnu
  - Build for amd64-musl
  
# When a tag is created, this step gets added.
- name: Create release on gitea
  image: plugins/gitea-release
  settings:
    api_key:
      from_secret: gitea_token
    base_url: https://code.kiers.eu
    checksum: sha256
    files: artifacts/*
  depends_on:
  - Show built artifacts
  when:
    event:
    - tag

Fin

You've made it! This is the end of the pipeline!

If this was useful for you, or you have comments, feel free to send me an email.