Jacob Kiers - developmentZola2022-12-30T00:00:00+00:00https://jacobkiers.net/tags/development/atom.xmlCross compiling Rust with Drone2022-12-30T00:00:00+00:002022-12-30T00:00:00+00:00Unknownhttps://jacobkiers.net/post/rust-cross-compilation-on-drone/<h2 id="why">Why</h2>
<p>For a while now, I have been working on a personal project called
<a rel="nofollow noreferrer" href="https://code.kiers.eu/newsletter-to-web/newsletter-to-web">newsletter2web</a>, which I run on a Raspberry PI. As the title
already indicates, it is written in <a rel="nofollow noreferrer" href="https://www.rust-lang.org/">Rust</a>.</p>
<p>Since compiling the project takes a while, especially on a Raspberry PI,
I decided to investigate how to make it build automatically.</p>
<p>That sounds like a job for continuous integration solution. But these tend
to use quite a bit of resources, even when not in use. That is wasteful
and I didn't want that.</p>
<p>So put together, I got the following requirements:</p>
<ul>
<li>A continuous integration solution, that</li>
<li>Would use little resources when not in use,</li>
<li>Integrates with Gitea, and</li>
<li>Is capable of cross-compiling Rust</li>
</ul>
<p>After a bit of digging, I found <a rel="nofollow noreferrer" href="https://www.drone.io/">Drone CI</a> to be a nice fit: it is fairly
lightweight, and easy to get running. Making it work well enough to cross
compile Rust took a while though.</p>
<p>This post describes the cross compilation, the <a href="https://jacobkiers.net/post/setting-up-drone-ci-with-autoscaler/">Drone and Gitea setup</a> is
described in the previous post.</p>
<h2 id="definitions">Definitions</h2>
<p>Since not everyone may be familiar with cross compilation, it is useful to
know some definitions.</p>
<ul>
<li>
<p>Host: the machine in which cross compilation is performed.</p>
</li>
<li>
<p>Target: the operating system and CPU architecture for which a project is
compiled.</p>
</li>
<li>
<p>Toolchain: the set of software required for compilation. This is at least a linker, but usually also a C compiler and target-specific source code are required.</p>
<p>When cross compiling, this is usually a different set of software than is required to compile for the host itself.</p>
</li>
</ul>
<h2 id="how-does-it-work">How does it work</h2>
<p>Rust has good support for cross compilation already out of the box.
However, in order to use it, the compilation toolchains need to be installed on the host. Also, the Rust project sometimes needs custom configuration as well, in order to be able to use this.</p>
<p>It can be tedious to do this manually. Therefore, we use <a rel="nofollow noreferrer" href="https://github.com/cross-rs/cross">cross</a>, which
is a project that uses pre-built Docker images for each target. Cross still
needs to have the <em>Rust</em> toolchain for each target installed though.</p>
<p>Since cross uses Docker, and compiling source code has the possibility of
executing arbitrary code on the host, a security boundary is required as
well. For that, a Docker in Docker setup is used.</p>
<p>Docker in docker means running the Docker service itself inside a Docker
container running on the host.</p>
<p>Since cross needs the Rust toolchain for each target, we will use a custom
Docker image to run it.</p>
<h2 id="the-docker-image">The Docker image</h2>
<p>This Docker image will have everything installed that is required to run
cross. So it contains the Rust toolchain for each target and the Docker
service for Docker in Docker.</p>
<p>This is the corresponding <code>Dockerfile</code>:</p>
<pre data-lang="Dockerfile" style="background-color:#2b303b;color:#c0c5ce;" class="language-Dockerfile "><code class="language-Dockerfile" data-lang="Dockerfile"><span style="color:#65737e;"># Base image. Change this to use another Rust version.
</span><span style="color:#b48ead;">FROM</span><span> rust:1.66-slim
</span><span>
</span><span style="color:#65737e;"># Install Docker (for docker-in-docker)
</span><span style="color:#b48ead;">RUN </span><span>apt-get update \
</span><span> && apt-get install -y ca-certificates curl gnupg \
</span><span> && mkdir -p /etc/apt/keyrings \
</span><span> && curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
</span><span> && echo "</span><span style="color:#a3be8c;">deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian bullseye stable</span><span>" >> /etc/apt/sources.list.d/docker.list \
</span><span> && apt-get update \
</span><span> && apt-get install -y docker-ce-cli \
</span><span> && apt-get clean
</span><span>
</span><span style="color:#65737e;"># Install cross v0.2.4
</span><span style="color:#b48ead;">RUN </span><span>mkdir -p /usr/local/cargo/bin && curl -L https://github.com/cross-rs/cross/releases/download/v0.2.4/cross-x86_64-unknown-linux-gnu.tar.gz | tar -xz -C /usr/local/cargo/bin
</span><span>
</span><span style="color:#65737e;"># (Optional) Pre-install rust targets
</span><span style="color:#65737e;"># This significantly reduces build time, at the cost of an increased image size
</span><span style="color:#b48ead;">RUN </span><span>rustup target add \
</span><span> aarch64-unknown-linux-gnu \
</span><span> aarch64-unknown-linux-musl \
</span><span> x86_64-pc-windows-gnu \
</span><span> x86_64-unknown-linux-gnu \
</span><span> x86_64-unknown-linux-musl \
</span><span> && rustup component add --target aarch64-unknown-linux-gnu rust-src \
</span><span> && rustup component add --target aarch64-unknown-linux-musl rust-src \
</span><span> && rustup component add --target x86_64-pc-windows-gnu rust-src \
</span><span> && rustup component add --target x86_64-unknown-linux-gnu rust-src \
</span><span> && rustup component add --target x86_64-unknown-linux-musl rust-src
</span></code></pre>
<p>Don't forget to push it to a registry, so it can be used by Drone later.</p>
<h2 id="the-drone-build-pipeline">The Drone build pipeline</h2>
<p>The build pipeline is where the real magic happens. I will use a
<code>.drone.yml</code> file here. In production I use a <a rel="nofollow noreferrer" href="https://jsonnet.org">Jsonnet</a> file to generate
this. You can find that <a rel="nofollow noreferrer" href="https://code.kiers.eu/newsletter-to-web/newsletter-to-web/src/tag/v0.2.3/.drone.jsonnet">in the newsletter-to-web repository</a>.</p>
<p>The pipeline is long, so it can be found at the
<a href="https://jacobkiers.net/post/rust-cross-compilation-on-drone/#full-build-pipeline">end of this post</a>.</p>
<h3 id="docker-in-docker-service">Docker in Docker service</h3>
<p>Before starting the builds, a running Docker agent is required. This is
handled by starting Docker in Docker as a service, and then waiting until
it is ready by a separate build step. All the cross compilation build steps
then wait for that to happen.</p>
<p>First, a shared volume is created for the Docker socket. That way, the agent
and all build steps can find each other.</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#65737e;"># Shared volume for the Docker in Docker service and build steps.
</span><span style="color:#65737e;"># Without this, cross won't be able to find a running Docker agent.
</span><span style="color:#bf616a;">volumes</span><span>:
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">temp</span><span>: {}
</span></code></pre>
<p>Then, the Docker in Docker service is defined. It uses the volume as
described. It will pull the Docker in Docker image and run it.</p>
<p>This requires <code>privileged</code> mode, because otherwise pulling images and
creating volumes fails.</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">services</span><span>:
</span><span>- </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">docker:dind
</span><span> </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">docker
</span><span> </span><span style="color:#bf616a;">privileged</span><span>: </span><span style="color:#d08770;">true </span><span style="color:#65737e;"># Docker in Docker requires this, sadly.
</span><span> </span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">volumes</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span></code></pre>
<p>Then, in the <code>steps</code>, the <code>Wait for Docker</code> step is defined. This will
try to run <code>docker image ls</code> until it succeeds. Once it is, it will print
information about the current Docker installation, and pull an image.
This is essentially a healt check.</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#65737e;"># Ensure Docker agent has started before continuing.
</span><span style="color:#bf616a;">steps</span><span>:
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Wait for Docker
</span><span> </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span> </span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">commands</span><span>:
</span><span> - </span><span style="color:#a3be8c;">mkdir artifacts
</span><span> - </span><span style="color:#a3be8c;">while ! docker image ls; do sleep 1; done
</span><span> - </span><span style="color:#a3be8c;">docker info
</span><span> - </span><span style="color:#a3be8c;">docker pull hello-world:latest
</span><span> </span><span style="color:#bf616a;">volumes</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span></code></pre>
<p>All other build steps can then wait until it is ready by depending on this
build step. Like so:</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">steps</span><span>:
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Build for amd64
</span><span> </span><span style="color:#bf616a;">depends_on</span><span>:
</span><span> - </span><span style="color:#a3be8c;">Wait for Docker
</span></code></pre>
<h3 id="cross-compilation">Cross compilation</h3>
<p>A cross compilation step is defined like so (under the <code>steps</code> definition):</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Build for windows-amd64
</span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span style="color:#bf616a;">environment</span><span>:
</span><span> </span><span style="color:#bf616a;">CROSS_REMOTE</span><span>: </span><span style="color:#d08770;">true
</span><span style="color:#bf616a;">depends_on</span><span>:
</span><span>- </span><span style="color:#a3be8c;">Wait for Docker
</span><span style="color:#bf616a;">volumes</span><span>:
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span><span style="color:#bf616a;">commands</span><span>:
</span><span>- </span><span style="color:#a3be8c;">cross build --release --target x86_64-pc-windows-gnu
</span><span>- </span><span style="color:#a3be8c;">cp target/x86_64-pc-windows-gnu/release/drone-test.exe artifacts/drone-test-windows-amd64.exe
</span><span>- </span><span style="color:#a3be8c;">rm -rf target/x86_64-pc-windows-gnu/release/*
</span></code></pre>
<p>It starts with defining the docker image to use <code>image: rust-dind-cross:1.66-full</code>. This is the images that was created before the section
<a href="https://jacobkiers.net/post/rust-cross-compilation-on-drone/#the-docker-image"><em>The Docker image</em></a>.</p>
<p>Then, it sets the environment:</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">environment</span><span>:
</span><span> </span><span style="color:#bf616a;">CROSS_REMOTE</span><span>: </span><span style="color:#d08770;">true
</span></code></pre>
<p>This tells cross that it is running in a remote docker, so it will copy
everything necessary for the build into the build image that it creates.</p>
<p>As described before, it depends on the Docker in Docker agent, so that
is defined as well, together with the shared volume:</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">depends_on</span><span>:
</span><span>- </span><span style="color:#a3be8c;">Wait for Docker
</span><span style="color:#bf616a;">volumes</span><span>:
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span></code></pre>
<p>Finally, the commands to run are listed:</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">commands</span><span>:
</span><span>- </span><span style="color:#a3be8c;">cross build --release --target x86_64-pc-windows-gnu
</span><span>- </span><span style="color:#a3be8c;">cp target/x86_64-pc-windows-gnu/release/drone-test.exe artifacts/drone-test-windows-amd64.exe
</span><span>- </span><span style="color:#a3be8c;">rm -rf target/x86_64-pc-windows-gnu/release/*
</span></code></pre>
<p>This will tell cross to build a release binary for the given target.
In order to do so, it will start its own Docker container, copies all
the source files into it, and then runs a normal Rust build.</p>
<p>The container started by cross is the container that contains the
required build toolchain for the specific target, as described in
the <a href="https://jacobkiers.net/post/rust-cross-compilation-on-drone/#definitions">definitions</a>.</p>
<p>When it is finished compiling, the binary is copied to an <code>artifacts/</code>
directory for later reference. It will now also gain a postfix with the
target it was build for. Otherwise, all binaries would be named the same
and would be overwritten.</p>
<p>Finally, the release directory is cleaned up in order to save disk space.</p>
<p>When all cross compilations have succeeded, the built artifacts are shown.
This step is not necessary, and you could do anything else with it.</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Show built artifacts
</span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span style="color:#bf616a;">commands</span><span>:
</span><span>- </span><span style="color:#a3be8c;">ls -lah artifacts
</span><span style="color:#bf616a;">depends_on</span><span>:
</span><span>- </span><span style="color:#a3be8c;">Build for arm64-gnu
</span><span>- </span><span style="color:#a3be8c;">Build for arm64-musl
</span><span>- </span><span style="color:#a3be8c;">Build for windows-amd64
</span><span>- </span><span style="color:#a3be8c;">Build for amd64-gnu
</span><span>- </span><span style="color:#a3be8c;">Build for amd64-musl
</span></code></pre>
<p>The interesting part here, is that it depends on all the cross compilation
build steps.</p>
<h3 id="when-tagged-create-a-gitea-release">When tagged, create a Gitea release</h3>
<p>When the pipeline is run for a tag, than as the very last step a new draft
release is created in Gitea. This can then be edited and published.</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Create release on gitea
</span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">plugins/gitea-release
</span><span style="color:#bf616a;">settings</span><span>:
</span><span> </span><span style="color:#bf616a;">api_key</span><span>:
</span><span> </span><span style="color:#bf616a;">from_secret</span><span>: </span><span style="color:#a3be8c;">gitea_token
</span><span> </span><span style="color:#bf616a;">base_url</span><span>: </span><span style="color:#a3be8c;">https://gitea.example.com
</span><span> </span><span style="color:#bf616a;">checksum</span><span>: </span><span style="color:#a3be8c;">sha256
</span><span> </span><span style="color:#bf616a;">files</span><span>: </span><span style="color:#a3be8c;">artifacts/*
</span><span> </span><span style="color:#bf616a;">draft</span><span>: </span><span style="color:#d08770;">true
</span><span style="color:#bf616a;">depends_on</span><span>:
</span><span>- </span><span style="color:#a3be8c;">Build for arm64-gnu
</span><span>- </span><span style="color:#a3be8c;">Build for arm64-musl
</span><span>- </span><span style="color:#a3be8c;">Build for windows-amd64
</span><span>- </span><span style="color:#a3be8c;">Build for amd64-gnu
</span><span>- </span><span style="color:#a3be8c;">Build for amd64-musl
</span><span style="color:#bf616a;">when</span><span>:
</span><span> </span><span style="color:#bf616a;">event</span><span>:
</span><span> - </span><span style="color:#a3be8c;">tag
</span></code></pre>
<p>This steps depends on a successful build of all cross compilation steps.
It uses a drone plugin called <a rel="nofollow noreferrer" href="https://plugins.drone.io/plugins/gitea-release">Gitea Release</a>.</p>
<p>It will:</p>
<ul>
<li>Upload all files in the <code>artifacts/</code> directory as assets.</li>
<li>Calculate the SHA256 sum of all artifacts, and add that as
a <code>sha256sum.txt</code> file to the assets.</li>
</ul>
<p>This only happens when the pipeline runs for a git tag, which is described
like this:</p>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">when</span><span>:
</span><span> </span><span style="color:#bf616a;">event</span><span>:
</span><span> - </span><span style="color:#a3be8c;">tag
</span></code></pre>
<h3 id="full-build-pipeline">Full build pipeline</h3>
<details>
<summary>Contents of .drone.yml</summary>
<span>
<pre data-lang="yml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yml "><code class="language-yml" data-lang="yml"><span style="color:#bf616a;">kind</span><span>: </span><span style="color:#a3be8c;">pipeline
</span><span style="color:#bf616a;">type</span><span>: </span><span style="color:#a3be8c;">docker
</span><span style="color:#bf616a;">name</span><span>: '</span><span style="color:#a3be8c;">Rust cross compilation</span><span>'
</span><span style="color:#bf616a;">platform</span><span>:
</span><span> </span><span style="color:#bf616a;">arch</span><span>: </span><span style="color:#a3be8c;">amd64
</span><span>
</span><span style="color:#bf616a;">image_pull_secrets</span><span>:
</span><span>- </span><span style="color:#a3be8c;">docker_private_repo
</span><span>
</span><span style="color:#65737e;"># Shared volume for the Docker in Docker service and build steps.
</span><span style="color:#65737e;"># Without this, cross won't be able to find a running Docker agent.
</span><span style="color:#bf616a;">volumes</span><span>:
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">temp</span><span>: {}
</span><span>
</span><span style="color:#bf616a;">services</span><span>:
</span><span>- </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">docker:dind
</span><span> </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">docker
</span><span> </span><span style="color:#bf616a;">privileged</span><span>: </span><span style="color:#d08770;">true </span><span style="color:#65737e;"># Docker in Docker requires this, sadly.
</span><span> </span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">volumes</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span><span>
</span><span style="color:#65737e;"># Ensure Docker agent has started before continuing.
</span><span style="color:#bf616a;">steps</span><span>:
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Wait for Docker
</span><span> </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span> </span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">commands</span><span>:
</span><span> - </span><span style="color:#a3be8c;">mkdir artifacts
</span><span> - </span><span style="color:#a3be8c;">while ! docker image ls; do sleep 1; done
</span><span> - </span><span style="color:#a3be8c;">docker info
</span><span> - </span><span style="color:#a3be8c;">docker pull hello-world:latest
</span><span> </span><span style="color:#bf616a;">volumes</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span><span>
</span><span style="color:#65737e;"># Build for aarch64-unknown-linux-gnu (arm64)
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Build for arm64-gnu
</span><span> </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span> </span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">environment</span><span>:
</span><span> </span><span style="color:#bf616a;">CROSS_REMOTE</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">commands</span><span>:
</span><span> - </span><span style="color:#a3be8c;">cross build --release --target aarch64-unknown-linux-gnu
</span><span> - </span><span style="color:#a3be8c;">cp target/aarch64-unknown-linux-gnu/release/drone-test artifacts/drone-test-arm64-gnu
</span><span> - </span><span style="color:#a3be8c;">rm -rf target/aarch64-unknown-linux-gnu/release/*
</span><span> </span><span style="color:#bf616a;">depends_on</span><span>:
</span><span> - </span><span style="color:#a3be8c;">Wait for Docker
</span><span> </span><span style="color:#bf616a;">volumes</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span><span>
</span><span style="color:#65737e;"># Build for aarch64-unknown-linux-musl (arm64 with Alpine linux)
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Build for arm64-musl
</span><span> </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span> </span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">environment</span><span>:
</span><span> </span><span style="color:#bf616a;">CROSS_REMOTE</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">commands</span><span>:
</span><span> - </span><span style="color:#a3be8c;">cross build --release --target aarch64-unknown-linux-musl
</span><span> - </span><span style="color:#a3be8c;">cp target/aarch64-unknown-linux-musl/release/drone-test artifacts/drone-test-arm64-musl
</span><span> - </span><span style="color:#a3be8c;">rm -rf target/aarch64-unknown-linux-musl/release/*
</span><span> </span><span style="color:#bf616a;">depends_on</span><span>:
</span><span> - </span><span style="color:#a3be8c;">Wait for Docker
</span><span> </span><span style="color:#bf616a;">volumes</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span><span>
</span><span style="color:#65737e;"># Build for x86_64-pc-windows-gnu
</span><span style="color:#65737e;"># This generates a binary that can run on Windows.
</span><span style="color:#bf616a;">-name</span><span>: </span><span style="color:#a3be8c;">Build for windows-amd64
</span><span> </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span> </span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">environment</span><span>:
</span><span> </span><span style="color:#bf616a;">CROSS_REMOTE</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">commands</span><span>:
</span><span> - </span><span style="color:#a3be8c;">cross build --release --target x86_64-pc-windows-gnu
</span><span> - </span><span style="color:#a3be8c;">cp target/x86_64-pc-windows-gnu/release/drone-test.exe artifacts/drone-test-windows-amd64.exe
</span><span> - </span><span style="color:#a3be8c;">rm -rf target/x86_64-pc-windows-gnu/release/*
</span><span> </span><span style="color:#bf616a;">depends_on</span><span>:
</span><span> - </span><span style="color:#a3be8c;">Wait for Docker
</span><span> </span><span style="color:#bf616a;">volumes</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span><span>
</span><span style="color:#65737e;"># Build for x86_64-unknown-linux-gnu (amd64)
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Build for amd64-gnu
</span><span> </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span> </span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">environment</span><span>:
</span><span> </span><span style="color:#bf616a;">CROSS_REMOTE</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">commands</span><span>:
</span><span> - </span><span style="color:#a3be8c;">cross build --release --target x86_64-unknown-linux-gnu
</span><span> - </span><span style="color:#a3be8c;">cp target/x86_64-unknown-linux-gnu/release/drone-test artifacts/drone-test-amd64-gnu
</span><span> - </span><span style="color:#a3be8c;">rm -rf target/x86_64-unknown-linux-gnu/release/*
</span><span> </span><span style="color:#bf616a;">depends_on</span><span>:
</span><span> - </span><span style="color:#a3be8c;">Wait for Docker
</span><span> </span><span style="color:#bf616a;">volumes</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span><span>
</span><span style="color:#65737e;"># Build for x86_64-unknown-linux-musl (amd64 with Alpine linux)
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Build for amd64-musl
</span><span> </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span> </span><span style="color:#bf616a;">pull</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">environment</span><span>:
</span><span> </span><span style="color:#bf616a;">CROSS_REMOTE</span><span>: </span><span style="color:#d08770;">true
</span><span> </span><span style="color:#bf616a;">commands</span><span>:
</span><span> - </span><span style="color:#a3be8c;">cross build --release --target x86_64-unknown-linux-musl
</span><span> - </span><span style="color:#a3be8c;">cp target/x86_64-unknown-linux-musl/release/drone-test artifacts/drone-test-amd64-musl
</span><span> - </span><span style="color:#a3be8c;">rm -rf target/x86_64-unknown-linux-musl/release/*
</span><span> </span><span style="color:#bf616a;">depends_on</span><span>:
</span><span> - </span><span style="color:#a3be8c;">Wait for Docker
</span><span> </span><span style="color:#bf616a;">volumes</span><span>:
</span><span> - </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">dockersock
</span><span> </span><span style="color:#bf616a;">path</span><span>: </span><span style="color:#a3be8c;">/var/run
</span><span>
</span><span style="color:#65737e;"># Show all built artifacts
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Show built artifacts
</span><span> </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">rust-dind-cross:1.66-full
</span><span> </span><span style="color:#bf616a;">commands</span><span>:
</span><span> - </span><span style="color:#a3be8c;">ls -lah artifacts
</span><span> </span><span style="color:#bf616a;">depends_on</span><span>:
</span><span> - </span><span style="color:#a3be8c;">Build for arm64-gnu
</span><span> - </span><span style="color:#a3be8c;">Build for arm64-musl
</span><span> - </span><span style="color:#a3be8c;">Build for windows-amd64
</span><span> - </span><span style="color:#a3be8c;">Build for amd64-gnu
</span><span> - </span><span style="color:#a3be8c;">Build for amd64-musl
</span><span>
</span><span style="color:#65737e;"># When a tag is created, this step gets added.
</span><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Create release on gitea
</span><span> </span><span style="color:#bf616a;">image</span><span>: </span><span style="color:#a3be8c;">plugins/gitea-release
</span><span> </span><span style="color:#bf616a;">settings</span><span>:
</span><span> </span><span style="color:#bf616a;">api_key</span><span>:
</span><span> </span><span style="color:#bf616a;">from_secret</span><span>: </span><span style="color:#a3be8c;">gitea_token
</span><span> </span><span style="color:#bf616a;">base_url</span><span>: </span><span style="color:#a3be8c;">https://code.kiers.eu
</span><span> </span><span style="color:#bf616a;">checksum</span><span>: </span><span style="color:#a3be8c;">sha256
</span><span> </span><span style="color:#bf616a;">files</span><span>: </span><span style="color:#a3be8c;">artifacts/*
</span><span> </span><span style="color:#bf616a;">depends_on</span><span>:
</span><span> - </span><span style="color:#a3be8c;">Show built artifacts
</span><span> </span><span style="color:#bf616a;">when</span><span>:
</span><span> </span><span style="color:#bf616a;">event</span><span>:
</span><span> - </span><span style="color:#a3be8c;">tag
</span></code></pre>
</span>
</details>
<h2 id="fin">Fin</h2>
<p>You've made it! This is the end of the pipeline!</p>
<p>If this was useful for you, or you have comments, feel free to <a href="mailto:hi@jacobkiers.net">send me an email</a>.</p>
Automatically scaling Drone CI with Gitea2022-12-17T00:00:00+00:002022-12-30T00:00:00+00:00Unknownhttps://jacobkiers.net/post/setting-up-drone-ci-with-autoscaler/<h2 id="why">Why</h2>
<p>For a while now, I have been working on a personal project called
<a rel="nofollow noreferrer" href="https://code.kiers.eu/newsletter-to-web/newsletter-to-web">newsletter2web</a>, which I run on a Raspberry PI. As the title
already indicates, it is written in <a rel="nofollow noreferrer" href="https://www.rust-lang.org/">Rust</a>.</p>
<p>Since compiling the project takes a while, especially on a Raspberry PI,
I decided to investigate how to make it build automatically.</p>
<p>That sounds like a job for continuous integration solution. But these tend
to use quite a bit of resources, even when not in use. That is wasteful
and I didn't want that.</p>
<p>So put together, I got the following requirements:</p>
<ul>
<li>A continuous integration solution, that</li>
<li>Would use little resources when not in use,</li>
<li>Integrates with Gitea, and</li>
<li>Is capable of cross-compiling Rust</li>
</ul>
<p>After a bit of digging, I found <a rel="nofollow noreferrer" href="https://www.drone.io/">Drone CI</a> to be a nice fit: it is fairly
lightweight, and easy to get running. Making it work well enough to cross
compile Rust took a while though.</p>
<p>This post describes the Drone and Gitea parts, <a href="https://jacobkiers.net/post/rust-cross-compilation-on-drone/">the Rust part is in a followup post</a>.</p>
<h2 id="the-components">The components</h2>
<p>In order to build everything, we need a few moving parts:</p>
<ul>
<li><a rel="nofollow noreferrer" href="https://gitea.io/en-us/">Gitea</a> for hosting the code.</li>
<li><a rel="nofollow noreferrer" href="https://www.drone.io/">Drone CI</a> for running the builds.</li>
<li><a rel="nofollow noreferrer" href="https://autoscale.drone.io/">Drone autoscaler</a> to ensure it only uses resources when necessary.</li>
<li><a rel="nofollow noreferrer" href="https://github.com/cross-rs/cross"><code>cross</code></a>: a cross compilation solution for Rust.</li>
</ul>
<h2 id="setting-up-drone-with-gitea">Setting up Drone with Gitea</h2>
<p>Setting up Drone is not that hard, and the <a rel="nofollow noreferrer" href="https://docs.drone.io/server/provider/gitea/">installation instructions</a> are clear.</p>
<p>Then, run it with docker using the <a rel="nofollow noreferrer" href="https://hub.docker.com/r/drone/drone">drone/drone:2</a> image, and ensure that
all these environment variables are passed through. Take care that the Drone
API is reachable from Gitea as well.</p>
<p>For me, the configuration came down to this:</p>
<pre data-lang="bash" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#65737e;"># Drone host configuration
</span><span style="color:#bf616a;">DRONE_SERVER_HOST</span><span>=</span><span style="color:#a3be8c;">drone-ci.kiers.eu
</span><span style="color:#bf616a;">DRONE_LOGS_DEBUG</span><span>=</span><span style="color:#a3be8c;">true
</span><span style="color:#bf616a;">DRONE_LOGS_PRETTY</span><span>=</span><span style="color:#a3be8c;">true
</span><span>
</span><span style="color:#65737e;"># RPC secret for build agents
</span><span style="color:#bf616a;">DRONE_RPC_SECRET</span><span>=<redacted>
</span><span>
</span><span style="color:#65737e;"># DB configuration
</span><span style="color:#bf616a;">DRONE_DATABASE_DATASOURCE</span><span>=</span><span style="color:#a3be8c;">postgres://drone:drone-password@db:5432/drone-db
</span><span style="color:#bf616a;">DRONE_DATABASE_SECRET</span><span>=<redacted>
</span><span>
</span><span style="color:#65737e;"># Gitea integration
</span><span style="color:#bf616a;">DRONE_GITEA_SERVER</span><span>=</span><span style="color:#a3be8c;">https://code.kiers.eu
</span><span style="color:#bf616a;">DRONE_GITEA_CLIENT_ID</span><span>=<redacted>
</span><span style="color:#bf616a;">DRONE_GITEA_CLIENT_SECRET</span><span>=<redacted>
</span><span>
</span><span style="color:#65737e;"># Enable initial admin user
</span><span style="color:#65737e;"># This needs to be the same username as in Gitea
</span><span style="color:#bf616a;">DRONE_USER_CREATE</span><span>=</span><span style="color:#a3be8c;">username:my-username,admin:true
</span><span>
</span><span style="color:#65737e;"># Enable Jsonnet support
</span><span style="color:#bf616a;">DRONE_JSONNET_ENABLED</span><span>=</span><span style="color:#a3be8c;">true
</span></code></pre>
<h2 id="setting-up-drone-autoscaler">Setting up Drone autoscaler</h2>
<p>During my research, I tried multiple approaches: first I used a simple Docker
runner on a static build server. But I wasn't happy to incur the costs of
something that is only used sporadically. Then I tried multiple different
solutions, such as the VM runner, the (deprecated) Digitalocean runner and
the <a rel="nofollow noreferrer" href="https://autoscale.drone.io/">autoscaler</a>. In the end, the last one was what I settled on.</p>
<p>It works by polling the Drone server api to see whether there are builds
pending and whether there is build capacity available. Then, it will scale
up or down the numer of servers that are required to handle these builds.
Crucially for me, it is possible to set minimum number of build servers to
zero, to ensure that they will all be destroyed when there is no need for
them anymore.</p>
<p>This does mean that one needs a place to host these build servers. Luckily,
the autoscaler has support for a number of different cloud providers. I
ended up using Digital Ocean.</p>
<p>First, a <a rel="nofollow noreferrer" href="https://docs.drone.io/manage/user/machine/">machine user</a> with administrative credentials needs to be created.
This needs to be done with the Drone CLI. Check the documentation for how
to do so.</p>
<p>The setup is fairly easy again, and also builds on environment variables:</p>
<pre data-lang="bash" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#65737e;">#!/usr/bin/bash
</span><span>
</span><span style="color:#65737e;"># Communication with the drone server
</span><span style="color:#bf616a;">DRONE_RPC_HOST</span><span>=</span><span style="color:#a3be8c;">drone-ci.kiers.eu
</span><span style="color:#bf616a;">DRONE_RPC_PROTO</span><span>=</span><span style="color:#a3be8c;">https
</span><span style="color:#bf616a;">DRONE_AUTOSCALER_USER_TOKEN</span><span>=<redacted> </span><span style="color:#65737e;"># Machine user token
</span><span>
</span><span style="color:#65737e;"># RPC Secret for the build agents that are installed on the build servers
</span><span style="color:#bf616a;">DRONE_RPC_SECRET</span><span>=<redacted>
</span><span>
</span><span style="color:#65737e;"># Digitalocean settings
</span><span style="color:#65737e;"># Get these from the DO Droplet Launch UI, in the URL bar
</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_SIZE</span><span>='</span><span style="color:#a3be8c;">s-4vcpu-8gb-amd</span><span>'
</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_IMAGE</span><span>='</span><span style="color:#a3be8c;">docker-20-04</span><span>'
</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_REGION</span><span>='</span><span style="color:#a3be8c;">fra1</span><span>'
</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_TOKEN</span><span>=<redacted>
</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_SSHKEY</span><span>=<redacted> </span><span style="color:#65737e;"># To log in to the build servers
</span><span>
</span><span style="color:#65737e;"># How often should the autoscaler check whether to scale up or down
</span><span style="color:#65737e;"># This then also is the time that it takes before a build server will
</span><span style="color:#65737e;"># be started when none is available, and therefore has a big influence
</span><span style="color:#65737e;"># on total build time.
</span><span style="color:#bf616a;">DRONE_INTERVAL</span><span>=</span><span style="color:#a3be8c;">30s
</span><span>
</span><span style="color:#65737e;"># Pool settings
</span><span style="color:#bf616a;">DRONE_POOL_MIN</span><span>=</span><span style="color:#a3be8c;">0 </span><span style="color:#65737e;"># This allows the pool to turn off everything
</span><span style="color:#bf616a;">DRONE_POOL_MAX</span><span>=</span><span style="color:#a3be8c;">4 </span><span style="color:#65737e;"># The maximum number of build servers to launch
</span><span>
</span><span style="color:#bf616a;">docker</span><span> run</span><span style="color:#bf616a;"> -d </span><span>\
</span><span style="color:#bf616a;"> --volume</span><span>=$(</span><span style="color:#bf616a;">pwd</span><span>)/data:/data \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_SERVER_PROTO=${</span><span style="color:#bf616a;">DRONE_RPC_PROTO</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_SERVER_HOST=${</span><span style="color:#bf616a;">DRONE_RPC_HOST</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_SERVER_TOKEN=${</span><span style="color:#bf616a;">DRONE_AUTOSCALER_USER_TOKEN</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_AGENT_TOKEN=${</span><span style="color:#bf616a;">DRONE_RPC_SECRET</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_POOL_MIN=${</span><span style="color:#bf616a;">DRONE_POOL_MIN</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_POOL_MAX=${</span><span style="color:#bf616a;">DRONE_POOL_MAX</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_DIGITALOCEAN_REGION=${</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_REGION</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_DIGITALOCEAN_TOKEN=${</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_TOKEN</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_DIGITALOCEAN_SIZE=${</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_SIZE</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_DIGITALOCEAN_SSHKEY=${</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_SSHKEY</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_DIGITALOCEAN_IMAGE=${</span><span style="color:#bf616a;">DRONE_DIGITALOCEAN_IMAGE</span><span>} \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_LOGS_TRACE=true \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_LOGS_DEBUG=true \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_LOGS_PRETTY=true \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_LOGS_COLOR=true \
</span><span style="color:#bf616a;"> --env</span><span>=DRONE_INTERVAL=${</span><span style="color:#bf616a;">DRONE_INTERVAL</span><span>} \
</span><span style="color:#bf616a;"> --publish</span><span>=8080:8080 \
</span><span style="color:#bf616a;"> --restart</span><span> always \
</span><span style="color:#bf616a;"> --name</span><span> drone-autoscaler \
</span><span> drone/autoscaler:latest
</span></code></pre>
<p>And that's it. Now it will scale up and down the build servers, as new
builds come in.</p>
<p>Also, take a look at all the <a rel="nofollow noreferrer" href="https://autoscale.drone.io/reference/">possible settings</a> for the autoscaler, as
there may be some options that are beneficial as well.</p>
HTML over DNS: Serving Blog Content Over DNS2021-08-17T00:00:00+00:002021-08-17T00:00:00+00:00Unknownhttps://jacobkiers.net/post/html-over-dns/<h2 id="what-s-up">What's up?</h2>
<p>The past few days, I did an experiment to <a rel="nofollow noreferrer" href="https://jacobkiers.net/hod/">serve blog content over DNS (see here)</a>.
That page contains mostly the same content ast this post, but it is served in a
completely different manner.</p>
<p>This works because of <a rel="nofollow noreferrer" href="https://en.wikipedia.org/wiki/DNS_over_HTTPS">DNS over HTTPS</a> for which there is an <a rel="nofollow noreferrer" href="https://developers.cloudflare.com/1.1.1.1/dns-over-https/json-format">API from
Cloudflare</a>.</p>
<p>Comments at <a rel="nofollow noreferrer" href="https://news.ycombinator.com/item?id=28218406">Hacker News</a>, apparently.</p>
<h2 id="how-it-works">How it works</h2>
<p>The DNS over HTTP API from Cloudflare is used to load the contents of that
page, essentially like this:</p>
<pre data-lang="js" style="background-color:#2b303b;color:#c0c5ce;" class="language-js "><code class="language-js" data-lang="js"><span style="color:#8fa1b3;">fetch</span><span>("</span><span style="color:#a3be8c;">https://cloudflare-dns.com/dns-query?ct=application/dns-json&type=TXT&name=post.hod.experiments.jacobkiers.net</span><span>");
</span></code></pre>
<p>The content itself is served over DNS, using CoreDNS, with these contents:</p>
<pre data-lang="coredns" style="background-color:#2b303b;color:#c0c5ce;" class="language-coredns "><code class="language-coredns" data-lang="coredns"><span>hod.experiments.jacobkiers.net.:53 {
</span><span> log
</span><span> auto hod.experiments.jacobkiers.net. {
</span><span> directory /etc/coredns/zones/
</span><span> reload 10s
</span><span> }
</span><span>}
</span></code></pre>
<p>This feeds into a zone file, which looks like this:</p>
<pre data-lang="zonefile" style="background-color:#2b303b;color:#c0c5ce;" class="language-zonefile "><code class="language-zonefile" data-lang="zonefile"><span>$TTL 5m ; Default TTL
</span><span>@ IN SOA experiments.jacobkiers.net. postmaster.jacobkiers.net. (
</span><span> 2021081612 ; serial
</span><span> 1h ; slave refresh interval
</span><span> 15m ; slave retry interval
</span><span> 1w ; slave copy expire time
</span><span> 1h ; NXDOMAIN cache time
</span><span> )
</span><span>
</span><span>$ORIGIN hod.experiments.jacobkiers.net.
</span><span>
</span><span>;
</span><span>; domain name servers
</span><span>;
</span><span>@ IN NS experiments.jacobkiers.net.
</span><span>
</span><span>
</span><span>;; START BLOG RECORDS
</span><span>; posts-2021-08-17-serving-blog-content-over-dns-md
</span><span>posts-2021-08-17-serving-blog-content-over-dns-md 60 IN TXT "t=text/markdown;c=3;h=2fd63f0f408ad1336283999d0487ced9;m=eyJ0aXRsZSI6IlNlcnZpbmcgYmxvZyBjb250ZW50IG92ZXIgRE5TIiwiZGF0ZSI6IjIwMjEtMDgtMTUiLCJhdXRob3IiOiJKYWNvYiBLaWVycyJ9"
</span><span>0.2fd63f0f408ad1336283999d0487ced9 60 IN TXT "WW91IG1pZ2h0IG5vdCBiZSBhYmxlIHRvIHNlZSBpdCBpbW1lZGlhdGVseSwgYnV0IHRoZSBjb250ZW50IG9mIHRoaXMgcGFnZSBpcyB2ZXJ2ZWQgb3ZlciBETlMuCgpUaGlzIHdvcmtzIGJlY2F1c2Ugb2YgdGhlIG5ldyBETlMtb3Zlci1IVFRQIHN1cHBvcnQsIHdoaWNoLCBhdCBsZWFzdCBhdCBDbG91ZGZsYXJlLCBhbHNvIGhhcy"
</span><span>1.2fd63f0f408ad1336283999d0487ced9 60 IN TXT "BhbiBBUEkuCgpUaGF0IEFQSSBpcyB1c2VkIHRvIGxvYWQgdGhlIGNvbnRlbnRzIG9mIHRoaXMgcGFnZSwgZXNzZW50aWFsbHkgbGlrZSB0aGlzOgoKYGBganMKZmV0Y2goImh0dHBzOi8vY2xvdWRmbGFyZS1kbnMuY29tL2Rucy1xdWVyeT9jdD1hcHBsaWNhdGlvbi9kbnMtanNvbiZ0eXBlPVRYVCZuYW1lPXBvc3QuaG9kLmV4cGVy"
</span><span>2.2fd63f0f408ad1336283999d0487ced9 60 IN TXT "aW1lbnRzLmphY29ia2llcnMubmV0Iik7CmBgYAoKUGxlYXNlIHNlZSB0aGUgW3NvdXJjZSBjb2RlXSBmb3IgdGhlIGRldGFpbHMgb2YgaG93IGl0IHdvcmtzLgoKW3NvdXJjZSBjb2RlXTogaHR0cHM6Ly9naXRodWIuY29tL2phY29ia2llcnMvaHRtbC1vdmVyLWRucwo="
</span></code></pre>
<p>These records are base64 encoded content, so when concatenated and decoded,
they give the content of the posts.</p>
<p>Please see the <a rel="nofollow noreferrer" href="https://github.com/jacobkiers/html-over-dns">source code</a> for the details.</p>
<h2 id="faq">FAQ</h2>
<h3 id="why-though">Why, though?</h3>
<p>In short: just because I could. It was one of those ideas I was wondering
idly about, and I decided to just try it.</p>
<h3 id="has-it-any-practical-use">Has it any practical use?</h3>
<p>It is not intended to have any. Since DNS records are fairly small, serving
images or something would quickly start consuming 100s of requests per second.
I wouldn't want to do that to Cloudflare 😄</p>
<p>It would be an interesting experiment to see how feasible that is though.</p>
Mockery: returning values and throwing exceptions2015-01-24T00:00:00+00:002015-01-24T00:00:00+00:00Unknownhttps://jacobkiers.net/post/multiple-return-values-with-mockery/<p>Last week, I had to write a piece of code that contains retry logic. Naturally, I want to test it. That proved trickier than expected.</p>
<p>The application code looks like this:</p>
<pre data-lang="php" style="background-color:#2b303b;color:#c0c5ce;" class="language-php "><code class="language-php" data-lang="php"><span style="color:#ab7967;"><?php
</span><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">Sender
</span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">protected </span><span>$</span><span style="color:#bf616a;">connection</span><span style="color:#eff1f5;">;
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">public function </span><span style="color:#8fa1b3;">send</span><span style="color:#eff1f5;">()
</span><span style="color:#eff1f5;"> {
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">success </span><span>= </span><span style="color:#d08770;">false</span><span style="color:#eff1f5;">;
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">i </span><span>= </span><span style="color:#d08770;">0</span><span style="color:#eff1f5;">;
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">do </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">i</span><span>++</span><span style="color:#eff1f5;">;
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">try </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">success </span><span>= $</span><span style="color:#bf616a;">this</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">doSend</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">i</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;"> } </span><span style="color:#b48ead;">catch</span><span style="color:#eff1f5;"> (</span><span style="color:#ebcb8b;">SenderException </span><span>$</span><span style="color:#bf616a;">e</span><span style="color:#eff1f5;">) {
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">success </span><span>= </span><span style="color:#d08770;">false</span><span style="color:#eff1f5;">;
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> } </span><span style="color:#b48ead;">while </span><span style="color:#eff1f5;">(</span><span>!$</span><span style="color:#bf616a;">success </span><span>&& $</span><span style="color:#bf616a;">i </span><span>< </span><span style="color:#d08770;">3</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">return </span><span>$</span><span style="color:#bf616a;">success</span><span style="color:#eff1f5;">;
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">protected function </span><span style="color:#8fa1b3;">doSend</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">data</span><span style="color:#eff1f5;">)
</span><span style="color:#eff1f5;"> {
</span><span style="color:#eff1f5;"> </span><span style="color:#65737e;">// Can throw SenderException
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">response </span><span>= $</span><span style="color:#bf616a;">this</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">connection</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">send</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">data</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">if </span><span style="color:#eff1f5;">(</span><span>'</span><span style="color:#a3be8c;">OK</span><span>' === $</span><span style="color:#bf616a;">response</span><span style="color:#eff1f5;">) {
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">return </span><span style="color:#d08770;">true</span><span style="color:#eff1f5;">;
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">return </span><span style="color:#d08770;">false</span><span style="color:#eff1f5;">;
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">}
</span></code></pre>
<p>I specifically want to test the retry logic, so I have to mock the ::doSend() method. Then I can simulate the different outcomes (returning true or false, or throwing a SenderException).</p>
<p>I use <a rel="nofollow noreferrer" href="https://github.com/mockery/mockery">Mockery</a> to do the real work. It is a great library. If you don't know it yet, please check it out. I will wait right here...</p>
<p>Now, since ::doSend() is a protected method, Mockery must be instructed to allow that.</p>
<p>So after the first try, I ended up with:</p>
<pre data-lang="php" style="background-color:#2b303b;color:#c0c5ce;" class="language-php "><code class="language-php" data-lang="php"><span style="color:#ab7967;"><?php
</span><span style="color:#b48ead;">public function </span><span style="color:#8fa1b3;">testItWillRetrySending</span><span>()
</span><span>{
</span><span> $</span><span style="color:#bf616a;">sender </span><span>= </span><span style="color:#ebcb8b;">M</span><span>::</span><span style="color:#bf616a;">mock</span><span>('</span><span style="color:#a3be8c;">Sender</span><span>');
</span><span> $</span><span style="color:#bf616a;">sender</span><span>-></span><span style="color:#bf616a;">shouldAllowMockingProtectedMethods</span><span>()
</span><span>
</span><span> $</span><span style="color:#bf616a;">sender</span><span>-></span><span style="color:#bf616a;">shouldReceive</span><span>('</span><span style="color:#a3be8c;">doSend</span><span>')
</span><span> -></span><span style="color:#bf616a;">andReturn</span><span>(</span><span style="color:#d08770;">false</span><span>, </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">Exception</span><span>());
</span><span>}
</span></code></pre>
<p>To my surprise, this did not work. Instead of throwing the exception, Mockery returns it. So my next try was this:</p>
<pre data-lang="php" style="background-color:#2b303b;color:#c0c5ce;" class="language-php "><code class="language-php" data-lang="php"><span style="color:#ab7967;"><?php
</span><span>$</span><span style="color:#bf616a;">sender</span><span>-></span><span style="color:#bf616a;">shouldReceive</span><span>('</span><span style="color:#a3be8c;">doSend</span><span>')
</span><span> -></span><span style="color:#bf616a;">andReturn</span><span>(</span><span style="color:#d08770;">false</span><span>)
</span><span> -></span><span style="color:#bf616a;">andThrow</span><span>(</span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">Exception</span><span>());
</span></code></pre>
<p>Another surprise: with this code, Mockery will always throw the exception, and ignore the first return value (false). After some debugging, I found out that Mockery just overwrites the return values in this case.</p>
<p>Fortunately, there is another way to return multiple return values: the ::andReturnUsing() method. It gives full control over the return values.</p>
<p>So I ended up with this testing code:</p>
<pre data-lang="php" style="background-color:#2b303b;color:#c0c5ce;" class="language-php "><code class="language-php" data-lang="php"><span style="color:#ab7967;"><?php
</span><span>$</span><span style="color:#bf616a;">return_value_generator </span><span>= </span><span style="color:#b48ead;">function </span><span>() {
</span><span> </span><span style="color:#b48ead;">static </span><span>$</span><span style="color:#bf616a;">counter </span><span>= </span><span style="color:#d08770;">0</span><span>;
</span><span>
</span><span> $</span><span style="color:#bf616a;">counter</span><span>++;
</span><span>
</span><span> </span><span style="color:#b48ead;">switch </span><span>($</span><span style="color:#bf616a;">counter</span><span>) {
</span><span> </span><span style="color:#b48ead;">case </span><span style="color:#d08770;">1</span><span>: </span><span style="color:#b48ead;">return </span><span style="color:#d08770;">false</span><span>;
</span><span> </span><span style="color:#b48ead;">case </span><span style="color:#d08770;">2</span><span>: </span><span style="color:#b48ead;">throw new </span><span style="color:#ebcb8b;">SenderException</span><span>();
</span><span> </span><span style="color:#b48ead;">case </span><span style="color:#d08770;">3</span><span>: </span><span style="color:#b48ead;">return </span><span style="color:#d08770;">true</span><span>;
</span><span> </span><span style="color:#b48ead;">default</span><span>: </span><span style="color:#b48ead;">throw new </span><span style="color:#ebcb8b;">Exception</span><span>("</span><span style="color:#a3be8c;">Should never reach this.</span><span>");
</span><span> }
</span><span>};
</span><span>
</span><span>$</span><span style="color:#bf616a;">sender </span><span>= </span><span style="color:#ebcb8b;">M</span><span>::</span><span style="color:#bf616a;">mock</span><span>('</span><span style="color:#a3be8c;">Sender</span><span>');
</span><span>
</span><span>$</span><span style="color:#bf616a;">sender</span><span>-></span><span style="color:#bf616a;">shouldAllowMockingProtectedMethods</span><span>()
</span><span> -></span><span style="color:#bf616a;">shouldReceive</span><span>('</span><span style="color:#a3be8c;">doSend</span><span>')
</span><span> -></span><span style="color:#bf616a;">andReturnUsing</span><span>($</span><span style="color:#bf616a;">return_value_generator</span><span>);
</span></code></pre>
<p>This works perfectly. It feels a bit like a hack though. So if you know a better way or have any other remarks, please let me know.</p>
Configure nginx with SSL and SPDY on Ubuntu 12.04 Precise2014-01-17T00:00:00+00:002014-01-17T00:00:00+00:00Unknownhttps://jacobkiers.net/post/configure-nginx-with-ssl-and-spdy-on-ubuntu-12-04-precise/<p>As an experiment, last night I decided to configure nginx with SSL and SPDY.
Finding information on how to get it working with Ubuntu 12.04 wasn't that
easy, so I decided to write the process down.</p>
<p>I will assume here that you already have a certificate. If you don’t, you can
get them from many vendors.</p>
<h2 id="add-the-repository">Add the repository</h2>
<p>First, you need to have version 1.4 or higher of nginx. You can get that on Ubuntu 12.04 by installing the nginx/stable ppa:</p>
<p><code>sudo add-apt-repository ppa:nginx/stable</code></p>
<h2 id="install-nginx">Install nginx</h2>
<p>Then, you will have to update apt and install nginx. If you already have
installed nginx, you will have to do a dist-upgrade.</p>
<p>So, when nginx is not yet installed:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>sudo apt-get update
</span><span>sudo apt-get install nginx
</span></code></pre>
<p>If nginx is already installed:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>sudo apt-get update
</span><span>sudo apt-get dist-upgrade
</span></code></pre>
<h2 id="install-certificates">Install certificates</h2>
<p>Now that the right version of nginx is installed, you first need to install
your certificate.</p>
<p>Put your private certificate into <code>/etc/ssl/private/</code>, and chmod it to <code>0400</code>:</p>
<p><code>sudo chmod 400 /etc/ssl/private/jacobkiers.net.key</code></p>
<p>Then, put your public certificate into <code>/etc/ssl/certs/</code>. In many cases, you
have to establish a certificate chain. That means that your web server sends
multiple certificates to the browser. Please check the documentation of the
certificate authority where you got your certificate.</p>
<p>You can do that as follows:</p>
<p><code>sudo sh -c 'cat /path/to/ca/certificate >> /etc/ssl/certs/your-certificate'</code></p>
<h2 id="update-site-configuration">Update site configuration</h2>
<p>Good, we’re almost done. The last thing you need to do is updating the nginx
site configuration. You just have to tell nginx that you want SSL and SPDY,
and where it can find the certificates.</p>
<p>Put this in the server block of your domain:</p>
<pre data-lang="nginx" style="background-color:#2b303b;color:#c0c5ce;" class="language-nginx "><code class="language-nginx" data-lang="nginx"><span>server {
</span><span> listen 443 ssl spdy;
</span><span>
</span><span> ssl_certificate /etc/ssl/certs/jacobkiers.net.crt;
</span><span> ssl_certificate_key /etc/ssl/private/jacobkiers.net.key;
</span><span>
</span><span> server_name jacobkiers.net;
</span><span>
</span><span> ####
</span><span> # All
</span><span> # other
</span><span> # configuration
</span><span> ####
</span><span>}
</span><span>
</span><span>And that’s it!
</span><span>
</span><span>Let’s check the configuration:
</span><span>
</span><span>sudo /etc/init.d/nginx configtest
</span><span>
</span><span>and when nginx indicates that everything is OK, restart it:
</span><span>
</span><span>sudo service nginx stop
</span><span>sudo service nginx start
</span><span>
</span><span>Your website should now be serving SSL over SPDY!
</span><span>Bonus: Serving Only SSL
</span><span>
</span><span>As I want to only serve my domain over SSL and only use the top-level domain, I had to add a little more configuration: two extra server blocks.
</span><span>
</span><span>server {
</span><span> # Rewrite non-https domains.
</span><span> listen 80;
</span><span>
</span><span> server_name jacobkiers.net www.jacobkiers.net;
</span><span> return 301 https://jacobkiers.net$request_uri;
</span><span>}
</span><span>
</span><span>server {
</span><span> # Rewrite https://www.jacobkiers.net/
</span><span> # to https://jacobkiers.net/
</span><span> listen 443 ssl spdy;
</span><span>
</span><span> ssl_certificate /etc/ssl/certs/jacobkiers.net.crt;
</span><span> ssl_certificate_key /etc/ssl/private/jacobkiers.net.key;
</span><span>
</span><span> server_name www.jacobkiers.net;
</span><span> return 301 https://jacobkiers.net$request_uri;
</span><span>}
</span></code></pre>
<p>And that is all.</p>
How to Dynamically Instantiate Classes in PHP2009-03-10T00:00:00+00:002009-03-10T00:00:00+00:00Unknownhttps://jacobkiers.net/post/how-to-dynamically-instantiate-classes-in-php/<p>For Annabel (the name of the product that I currently work on), I need to
be able to dynamically instantiate classes, based on options in a
configuration file (Zend_Config_Ini). However, some of these classes have
constructors (some with required options), others don’t even have a constructor.
I first tried to instantiate them with call_user_func_array, like this:</p>
<pre data-lang="php" style="background-color:#2b303b;color:#c0c5ce;" class="language-php "><code class="language-php" data-lang="php"><span style="color:#ab7967;"><?php
</span><span>$</span><span style="color:#bf616a;">datatype_options </span><span>= </span><span style="color:#96b5b4;">explode</span><span>('</span><span style="color:#a3be8c;">,</span><span>', $</span><span style="color:#bf616a;">global_config</span><span>-></span><span style="color:#bf616a;">customer</span><span>-></span><span style="color:#bf616a;">data</span><span>-></span><span style="color:#bf616a;">field</span><span>->$</span><span style="color:#bf616a;">key</span><span>-></span><span style="color:#bf616a;">type</span><span>);
</span><span>$</span><span style="color:#bf616a;">datatype_class </span><span>= </span><span style="color:#96b5b4;">array_shift</span><span>($</span><span style="color:#bf616a;">datatype_options</span><span>);
</span><span>$</span><span style="color:#bf616a;">class </span><span>= </span><span style="color:#96b5b4;">call_user_func_array</span><span>(</span><span style="color:#96b5b4;">array</span><span>($</span><span style="color:#bf616a;">datatype_class</span><span>, '</span><span style="color:#a3be8c;">__construct</span><span>'), $</span><span style="color:#bf616a;">datatype_options</span><span>);
</span></code></pre>
<p>This didn’t work, since some classes don’t have a constructor, and they were
not static. Then, I came up with another solution: use a generic static
method getInstance() to do it, like this:</p>
<pre data-lang="php" style="background-color:#2b303b;color:#c0c5ce;" class="language-php "><code class="language-php" data-lang="php"><span style="color:#ab7967;"><?php
</span><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">Test </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">public static function </span><span style="color:#8fa1b3;">getInstance</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">classname</span><span style="color:#eff1f5;">)
</span><span style="color:#eff1f5;"> {
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">return new </span><span>$</span><span style="color:#bf616a;">classname</span><span style="color:#eff1f5;">();
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">}
</span><span>
</span><span>$</span><span style="color:#bf616a;">datatype_options </span><span>= </span><span style="color:#96b5b4;">explode</span><span>('</span><span style="color:#a3be8c;">,</span><span>', $</span><span style="color:#bf616a;">global_config</span><span>-></span><span style="color:#bf616a;">customer</span><span>-></span><span style="color:#bf616a;">data</span><span>-></span><span style="color:#bf616a;">field</span><span>->$</span><span style="color:#bf616a;">key</span><span>-></span><span style="color:#bf616a;">type</span><span>);
</span><span>$</span><span style="color:#bf616a;">datatype_class </span><span>= </span><span style="color:#96b5b4;">array_shift</span><span>($</span><span style="color:#bf616a;">datatype_options</span><span>);
</span><span>$</span><span style="color:#bf616a;">class </span><span>= </span><span style="color:#96b5b4;">call_user_func_array</span><span>(</span><span style="color:#96b5b4;">array</span><span>('</span><span style="color:#a3be8c;">Test</span><span>', '</span><span style="color:#a3be8c;">getInstance</span><span>'), $</span><span style="color:#bf616a;">datatype_options</span><span>);
</span></code></pre>
<p>This worked, as long as I don’t need any constructor parameters. But for
some classes, I <em>do</em> need them. So I thought some more, and came to the
conclusion that I would need the Reflection API to do what I want. Finally,
I implemented the following solution, which works perfectly. It could
probably be cleaned up a little, but it basically works:</p>
<pre data-lang="php" style="background-color:#2b303b;color:#c0c5ce;" class="language-php "><code class="language-php" data-lang="php"><span style="color:#ab7967;"><?php
</span><span style="color:#b48ead;">class </span><span style="color:#ebcb8b;">Annabel_Data
</span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#65737e;">/**
</span><span style="color:#65737e;"> * Creates instances of classes in the Annabel_Data package.
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * To use this, provide arguments in the order you would need them
</span><span style="color:#65737e;"> * in the instantiated classes, with as the first argument the name
</span><span style="color:#65737e;"> * of the class to instantiate.
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * </span><span style="color:#b48ead;">@param</span><span style="color:#65737e;"> $arguments An array with arguments
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * </span><span style="color:#b48ead;">@return</span><span style="color:#65737e;"> Annabel_Data_Abstract
</span><span style="color:#65737e;"> */
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">public static function </span><span style="color:#8fa1b3;">factory</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">arguments</span><span style="color:#eff1f5;">)
</span><span style="color:#eff1f5;"> {
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">class_name </span><span>= </span><span style="color:#96b5b4;">array_shift</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">arguments</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">class_name </span><span>= '</span><span style="color:#a3be8c;">Annabel_Data_</span><span>' . </span><span style="color:#96b5b4;">ucfirst</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">class_name</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;"> </span><span style="color:#ebcb8b;">Zend_Loader</span><span style="color:#eff1f5;">::</span><span style="color:#bf616a;">loadClass</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">class_name</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">reflector </span><span>= </span><span style="color:#b48ead;">new </span><span style="color:#ebcb8b;">ReflectionClass</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">class_name</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">if </span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">reflector</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">isInstantiable</span><span style="color:#eff1f5;">()) {
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">constructor </span><span>= $</span><span style="color:#bf616a;">reflector</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">getConstructor</span><span style="color:#eff1f5;">();
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">if </span><span style="color:#eff1f5;">(</span><span style="color:#96b5b4;">is_null</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">constructor</span><span style="color:#eff1f5;">)) {
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">return </span><span>$</span><span style="color:#bf616a;">reflector</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">newInstance</span><span style="color:#eff1f5;">();
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">params </span><span>= $</span><span style="color:#bf616a;">constructor</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">getNumberOfParameters</span><span style="color:#eff1f5;">();
</span><span style="color:#eff1f5;"> </span><span>$</span><span style="color:#bf616a;">req_params </span><span>= $</span><span style="color:#bf616a;">constructor</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">getNumberOfRequiredParameters</span><span style="color:#eff1f5;">();
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">if </span><span style="color:#eff1f5;">(</span><span style="color:#96b5b4;">count</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">arguments</span><span style="color:#eff1f5;">) </span><span>> $</span><span style="color:#bf616a;">req_params</span><span style="color:#eff1f5;">) {
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">throw new </span><span style="color:#ebcb8b;">Annabel_Data_Exception</span><span style="color:#eff1f5;">(</span><span>'</span><span style="color:#a3be8c;">Please provide </span><span>' . $</span><span style="color:#bf616a;">req_params </span><span>. ' </span><span style="color:#a3be8c;">parameters to instantiate class </span><span>' . $</span><span style="color:#bf616a;">class_name</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">if </span><span style="color:#eff1f5;">(</span><span style="color:#d08770;">0 </span><span>> $</span><span style="color:#bf616a;">params</span><span style="color:#eff1f5;">) {
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">return </span><span>$</span><span style="color:#bf616a;">reflector</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">newInstanceArgs</span><span style="color:#eff1f5;">(</span><span>$</span><span style="color:#bf616a;">arguments</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;"> } </span><span style="color:#b48ead;">else </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">return </span><span>$</span><span style="color:#bf616a;">reflector</span><span style="color:#eff1f5;">-></span><span style="color:#bf616a;">newInstance</span><span style="color:#eff1f5;">();
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> } </span><span style="color:#b48ead;">else </span><span style="color:#eff1f5;">{
</span><span style="color:#eff1f5;"> </span><span style="color:#b48ead;">throw new </span><span style="color:#ebcb8b;">Annabel_Data_Exception</span><span style="color:#eff1f5;">(</span><span>"</span><span style="color:#a3be8c;">Could not instantiate a class of type '</span><span>" . $</span><span style="color:#bf616a;">class_name </span><span>. "</span><span style="color:#a3be8c;">'</span><span>"</span><span style="color:#eff1f5;">);
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;"> }
</span><span style="color:#eff1f5;">}
</span><span>
</span><span>$</span><span style="color:#bf616a;">datatype_options </span><span>= </span><span style="color:#96b5b4;">explode</span><span>('</span><span style="color:#a3be8c;">,</span><span>', $</span><span style="color:#bf616a;">global_config</span><span>-></span><span style="color:#bf616a;">customer</span><span>-></span><span style="color:#bf616a;">data</span><span>-></span><span style="color:#bf616a;">field</span><span>->$</span><span style="color:#bf616a;">key</span><span>-></span><span style="color:#bf616a;">type</span><span>);
</span><span>$</span><span style="color:#bf616a;">class </span><span>= </span><span style="color:#ebcb8b;">Annabel_Data</span><span>::</span><span style="color:#bf616a;">factory</span><span>($</span><span style="color:#bf616a;">datatype_options</span><span>);
</span></code></pre>
<p>This works perfectly, for each case I need it (and tested it 😉). I should
add some more exception checking and things like that, but those are minor
details. Overall, I'm pretty happy with the solution!</p>
40x Speedup With iconv And PHP2009-03-03T00:00:00+00:002009-03-03T00:00:00+00:00Unknownhttps://jacobkiers.net/post/40x-speedup-with-iconv-and-php/<p>For our product Annabel, we have to clean up the data our customers provide
us. Because this is a fully automated process, we are unable to give direct
feedback and have them fix their input. Therefore, I need a means to clean
the data on our end, so we can process it.</p>
<p>Since we don’t need to support any unicode stuff, we can stick with just
plain ASCII. That’s a very safe approach, which will reduce the chances of
failure greatly. To convert the UTF-8 (Unicode) input into ASCII data, we use
the <a rel="nofollow noreferrer" href="http://www.gnu.org/software/libtool/manual/libc/Generic-Charset-Conversion.html#Generic-Charset-Conversion">iconv method from the GNU C Library</a> in combination with PHP.</p>
<p>The default iconv implementation in PHP has two caveats: it stops when a
string cannot be converted, and it prints a question mark when it does not
have an equivalent (or transliterated) character in the destination character
set. To overcome this problem, I used to just convert every single character
with the <a rel="nofollow noreferrer" href="https://www.php.net/manual/en/function.iconv.php">PHP iconv function</a>, which gave me a throughput of about 250KiB/sec,
using the following code:</p>
<pre data-lang="php" style="background-color:#2b303b;color:#c0c5ce;" class="language-php "><code class="language-php" data-lang="php"><span style="color:#ab7967;"><?php
</span><span style="color:#65737e;">/**
</span><span style="color:#65737e;">* Replaces special characters with their ASCII equivalents.
</span><span style="color:#65737e;">*
</span><span style="color:#65737e;">* This function uses iconv to replace each seperate character with its
</span><span style="color:#65737e;">* ASCII equivalent, using the ASCII//TRANSLIT option. However, this makes
</span><span style="color:#65737e;">* the function very slow: max throughput is about 150KiB/sec.
</span><span style="color:#65737e;">*
</span><span style="color:#65737e;">* </span><span style="color:#b48ead;">@param</span><span style="color:#65737e;"> string $line
</span><span style="color:#65737e;">* </span><span style="color:#b48ead;">@return</span><span style="color:#65737e;"> string
</span><span style="color:#65737e;">*/
</span><span style="color:#b48ead;">protected function </span><span style="color:#8fa1b3;">_convertSpecialChars</span><span>($</span><span style="color:#bf616a;">line</span><span>) {
</span><span> </span><span style="color:#b48ead;">if </span><span>(</span><span style="color:#96b5b4;">empty</span><span>($</span><span style="color:#bf616a;">line</span><span>)) </span><span style="color:#b48ead;">return </span><span>'';
</span><span>
</span><span> $</span><span style="color:#bf616a;">new_line </span><span>= "";
</span><span>
</span><span> </span><span style="color:#65737e;">/*
</span><span style="color:#65737e;"> * This potentially could be a very long string, so don't split the line
</span><span style="color:#65737e;"> * in separate tokens, for that would tak way too much memory.
</span><span style="color:#65737e;"> */
</span><span> $</span><span style="color:#bf616a;">line_length </span><span>= </span><span style="color:#96b5b4;">strlen</span><span>($</span><span style="color:#bf616a;">line</span><span>);
</span><span>
</span><span> </span><span style="color:#b48ead;">for </span><span>($</span><span style="color:#bf616a;">x </span><span>= </span><span style="color:#d08770;">0</span><span>; $</span><span style="color:#bf616a;">x </span><span>> $</span><span style="color:#bf616a;">line_length</span><span>; $</span><span style="color:#bf616a;">x</span><span>++) {
</span><span> $</span><span style="color:#bf616a;">old_char </span><span>= </span><span style="color:#96b5b4;">substr</span><span>($</span><span style="color:#bf616a;">line</span><span>, $</span><span style="color:#bf616a;">x</span><span>, </span><span style="color:#d08770;">1</span><span>);
</span><span>
</span><span> </span><span style="color:#65737e;">/*
</span><span style="color:#65737e;"> * Use iconv to replace the other special characters.
</span><span style="color:#65737e;"> * If iconv can't convert it (and so returns '?'), just skip
</span><span style="color:#65737e;"> * the character, for it probably is something malicious and
</span><span style="color:#65737e;"> * there's probably no need to keep it anyway.
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> * Beware of the edge case if the original character is a
</span><span style="color:#65737e;"> * question mark itself.
</span><span style="color:#65737e;"> */
</span><span> $</span><span style="color:#bf616a;">char </span><span>= </span><span style="color:#96b5b4;">iconv</span><span>('</span><span style="color:#a3be8c;">UTF-8</span><span>', '</span><span style="color:#a3be8c;">ASCII//TRANSLIT</span><span>', $</span><span style="color:#bf616a;">old_char</span><span>);
</span><span> </span><span style="color:#b48ead;">if </span><span>( ('</span><span style="color:#a3be8c;">?</span><span>' != $</span><span style="color:#bf616a;">char</span><span>) && ('</span><span style="color:#a3be8c;">?</span><span>' != $</span><span style="color:#bf616a;">old_char</span><span>) ) $</span><span style="color:#bf616a;">new_line </span><span>.= $</span><span style="color:#bf616a;">char</span><span>;
</span><span> }
</span><span>
</span><span> </span><span style="color:#b48ead;">return </span><span>$</span><span style="color:#bf616a;">new_line</span><span>;
</span><span>}
</span></code></pre>
<p>However, I was not satisfied with this, so I looked up the man page of the iconv
version of GNU C Library. I supposed PHP was internally using this one, so that
seemed a natural action. In that man-page I found the IGNORE option, which just
skips any character which cannot be converted or transliterated. That was exactly
what I wanted. So I tried that with the PHP function as well, and it worked.</p>
<p>Instead of converting every single character, I can now convert a whole file at
once, which gave me a throughput of 11MiB/sec. The caveat, of course, is that I
have to use the GNU C Library iconv, with a version the same (or greater than)
the current one, to avoid compatibility problems. However, that’s a price I’m
surely willing to pay. The new code is like this:</p>
<pre data-lang="php" style="background-color:#2b303b;color:#c0c5ce;" class="language-php "><code class="language-php" data-lang="php"><span style="color:#ab7967;"><?php
</span><span style="color:#65737e;">/**
</span><span style="color:#65737e;">* Replaces special characters with their ASCII equivalents.
</span><span style="color:#65737e;">*
</span><span style="color:#65737e;">* This function uses iconv to replace each seperate character with its
</span><span style="color:#65737e;">* ASCII equivalent, using the ASCII//TRANSLIT,IGNORE option. Throughput
</span><span style="color:#65737e;">* is measured at about 11MiB/sec.
</span><span style="color:#65737e;">*
</span><span style="color:#65737e;">* WARNING: Using the extra IGNORE option only works with a recent
</span><span style="color:#65737e;">* GNU libc iconv, so be very picky about which iconv to use! This is an
</span><span style="color:#65737e;">* undocumented feature, which is not supported by default and is not
</span><span style="color:#65737e;">* listed in the PHP manual!
</span><span style="color:#65737e;">*
</span><span style="color:#65737e;">* </span><span style="color:#b48ead;">@param</span><span style="color:#65737e;"> string $line
</span><span style="color:#65737e;">* </span><span style="color:#b48ead;">@return</span><span style="color:#65737e;"> string
</span><span style="color:#65737e;">*/
</span><span style="color:#b48ead;">protected function </span><span style="color:#8fa1b3;">_convertSpecialChars</span><span>($</span><span style="color:#bf616a;">line</span><span>)
</span><span>{
</span><span> </span><span style="color:#65737e;">/*
</span><span style="color:#65737e;"> * Check whether we have the right version of iconv
</span><span style="color:#65737e;"> */
</span><span> </span><span style="color:#b48ead;">if </span><span>( ('</span><span style="color:#a3be8c;">glibc</span><span>' !== ICONV_IMPL) || (</span><span style="color:#d08770;">true </span><span>== </span><span style="color:#96b5b4;">version_compare</span><span>(ICONV_VERSION, '</span><span style="color:#a3be8c;">2.8.90</span><span>', '</span><span style="color:#a3be8c;">></span><span>')) ) {
</span><span> </span><span style="color:#b48ead;">throw new </span><span style="color:#ebcb8b;">Exception</span><span>('</span><span style="color:#a3be8c;">Please use the glibc iconv, version 2.8.90 or higher</span><span>');
</span><span> }
</span><span>
</span><span> </span><span style="color:#65737e;">/*
</span><span style="color:#65737e;"> * Use iconv for speed and glory
</span><span style="color:#65737e;"> * We use the ASCII//TRANSLIT,IGNORE option to replace the string
</span><span style="color:#65737e;"> * with its ASCII transliterated equivalent. If there's no ASCII
</span><span style="color:#65737e;"> * equivalent, the IGNORE option makes sure the character is just
</span><span style="color:#65737e;"> * thrown away, which is exactly what I want.
</span><span style="color:#65737e;"> *
</span><span style="color:#65737e;"> */
</span><span> $</span><span style="color:#bf616a;">new_line </span><span>= </span><span style="color:#96b5b4;">iconv</span><span>('</span><span style="color:#a3be8c;">UTF-8</span><span>', '</span><span style="color:#a3be8c;">ASCII//TRANSLIT,IGNORE</span><span>', $</span><span style="color:#bf616a;">line</span><span>);
</span><span>
</span><span> </span><span style="color:#b48ead;">return </span><span>$</span><span style="color:#bf616a;">new_line</span><span>;
</span><span>}
</span></code></pre>
<p>I guess I don’t need to give further comments on this code example 😉</p>
<p>If you have questions or comments, you could <a href="mailto:hi@jacobkiers.net?subject=Comment%20on%20article%20about%20iconv">drop me a line</a>.</p>
Automatically add user to Ubuntu Linux and set password2008-12-23T00:00:00+00:002008-12-23T00:00:00+00:00Unknownhttps://jacobkiers.net/post/automatically-add-user-to-ubuntu-linux-and-set-password/<p>I was wondering how to create a user on my system without requiring a
prompt or CLI access. The following script is the result:</p>
<pre data-lang="bash" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#65737e;">#!/bin/bash
</span><span style="color:#bf616a;">UADD</span><span>=</span><span style="color:#a3be8c;">/usr/sbin/useradd
</span><span style="color:#bf616a;">OPENSSL</span><span>=</span><span style="color:#a3be8c;">/usr/bin/openssl
</span><span style="color:#bf616a;">SHELL</span><span>=</span><span style="color:#a3be8c;">/bin/bash
</span><span>
</span><span style="color:#65737e;"># Generate 12 characters long password and print it.
</span><span style="color:#bf616a;">PASS</span><span>=`$</span><span style="color:#bf616a;">OPENSSL</span><span> rand</span><span style="color:#bf616a;"> -base64</span><span> 12`
</span><span style="color:#96b5b4;">echo </span><span>$</span><span style="color:#bf616a;">PASS
</span><span>
</span><span style="color:#65737e;"># Make the password usable for the useradd utility
</span><span style="color:#bf616a;">PASS</span><span>=`</span><span style="color:#bf616a;">mkpasswd </span><span>$</span><span style="color:#bf616a;">PASS</span><span>`
</span><span>
</span><span style="color:#65737e;"># Create the user
</span><span>$</span><span style="color:#bf616a;">UADD -s </span><span>$</span><span style="color:#bf616a;">SHELL -m </span><span>$</span><span style="color:#bf616a;">1 -p </span><span>$</span><span style="color:#bf616a;">PASS
</span></code></pre>
PHP subversion post-commit hook2008-06-11T00:00:00+00:002008-06-11T00:00:00+00:00Unknownhttps://jacobkiers.net/post/php-subversion-post-commit-hook/<p>Some time ago I’ve written a post-commit hook for Subversion in PHP, to make it easier to track code changes via email.</p>
<p>I based it on <a rel="nofollow noreferrer" href="https://web.archive.org/web/20150906135345/https://elliotth.blogspot.com/2005/02/better-subversion-post-commit-hook.html">another hook</a>, written in Ruby. Because it had no license attached to it, and because I had to rewrite it myself, I decided to license it under the new BSD license.</p>
<p>You can find the hook here: <a rel="nofollow noreferrer" href="https://gist.github.com/jacobkiers/2a035367a56be1428487">commit-email-color.php</a>. It has the new BSD license attached to it.</p>
<p>Enjoy it!</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>2009-06-17: fixed a bug: now displays the full commit message instead of just the first line.
</span><span>2008-06-13: added an option to disable color-code the diffs.
</span></code></pre>