IoT Platform Design Doc: Distributed OurCI

Marko Mikulicic, Jul 28, 2016 10:47 AM

Our engineers share the background to our Continuous Integration (CI) process in this design doc. Internally, the team decided to nickname it OurCI and you’ll see it referred to as such in the design doc below. 

As always, the below is the original design doc our engineering team used to outline their thought process for the development part of Mongoose IoT Platform. For additional context we’ve added a few blue comments for you. Still included are all our internal puns (sorry, about those!), and if anything is unclear and you want to dive deeper, head over to the Forum and ask your question there.

Distributed OurCI

Objective

Design OurCI service that can distribute builds to remote agents. Some agents will perform builds, some other agents will flash devices and run Hardware (HW) tests. Job execution is parallel.

Non goals: defining the exact behavior of HW tests.

Background

Our current CI is slow and cannot run HW tests.

Parallelising builds on a multicore host is not easy in the current setup. It’s not just about adding -j32 to our make invocation. We need to run builds with different toolchains that are distributed in different docker images and we need to be able to build things in parallel but still have a readable output in case of build failures.

The core of our cloud backend is ready enough to serve as a basis for our continuous integration tool. Using it will offer us the opportunity to dogfood our infrastructure. This will affect some choices here. We also have a drive towards avoiding over-engineering. The proposed solution hopefully contains a balanced tradeoff between using our cloud infrastructure for its own sake and being simple enough so it can be relied upon even while we are doing massive changes on the very system we're testing.

Overview

  1. Implement generic Pub Sub service modeled after Google Cloud Pub Sub, and already designed some time ago but not yet implemented.
  2. Agents will talk clubby, execute shell scripts (or whatever) and report back output and status.
  3. Current builds’ scripts will be incrementally adapted to take advantage of the ability to spawn child builds.
  4. HW tests are just shell scripts that run on agents running on hosts that are connected to devices.

The ability to spawn new build tasks from a shell script allows us to build simple workflows.

There is no support for fancy things such a join nodes, node dependencies or anyway defining a workflow in a declarative way: the execute tasks are defined at runtime by a seed master script.

Design_Doc_OurCI_1.png

Green boxes here are messages published via Pub Sub. Dotted lines are refer to tasks executed by.

Design narrative

Distribution protocol is based on our Pub Sub service.

The OurCI master process will listen to GitHub events and spawn master builds by publishing a build request message.

A master build script will spawn child builds by publishing build request messages to more specific topics.

Agents are subscribers to pub sub topics. The generic agent listens to a pub sub topic and runs a subprocess (e.g. shell script). The script in the subprocess will perform a task and its execution status will be sent back to the OurCI master. Script output will be streamed to the Log Store service.

Each agent can perform one task at time. It will poll for Pub Sub messages and once received it will periodically acknowledge it until the subprocess is done or a build timeout occurs. If the Pub Sub message times out before the agent explicitly replies (either with success or failure, including build timeout failures), the Pub Sub services will make the message available for another agent to grab. We thus distinguish a build script timing out and an agent failing to communicate with the Pub Sub service.

The number of agent processes running on a given machine defines the parallelism we can achieve on that host. An agent can poll for multiple topics, but will only execute one at a time.

Agent scripts can spawn docker containers that perform the build or execute Mongoose Flashing Tool or esptool to flash a device.

The agent will communicate with the build script via environment variables. The build request type will thus be available to the build script. A single build script will thus be able to run the correct build step based on the actual build request, e.g. “run v7 tests”, “build Mongoose IoT Platform ESP8266”.

The Pub Sub topic is used to route the build request to an agent capable of understanding that particular build request.

Some agent scripts will store the build artefacts required by subsequent build steps. For example, a firmware (FW) will be built by a cross compiler running in a docker container on the cloud, stored in the blobstore and later fetched by a HW agent and flashed on a the device.

Not all build artefacts are necessarily stored in the blobstore. For example, build scripts that create docker images can publish them to the docker registry.

Detailed Design

Topics and build environments

A Pub Sub topic is just a string, telling which build environment is expected from the agent subscribing to that topic:

  1. build/docker
  2. test/esp-12e
  3. test/cc3200

In future, we can add more build environments, like a native windows build or osx build machines.

build/docker

This is our current build environment: a host with srcfs available at /data/srcfs, ssh credentials to pull the latest commits from GitHub and the possibility to run docker containers as build steps.

Our entire current `ourci` build script can be ran by an agent providing the `build:docker` build environment.

test/esp-12e

This build environment runs a script on a host with an attached esp-12e device with 4M flash and Mongoose Flashing Tool or esptool installed.

The serial port and clubby credentials for fetching the FW and test scripts via blobstore are available to the script via environment variables.

Esp-12e devices will come with some hardware attached. For simplicity, we currently provide no way to negotiate the actual set of attached hardware, hence, the test script and the environment will have to match.

The build environment doesn’t come with a git checkout nor with the credentials to fetch one. It totally depends on previous build tasks to provide it with all necessary files to flash the firmware and run a test script.

The test downloaded script will actually perform the flashing, boot the device, wait for WebDAV, upload a JS test script and listen for test results from the device. Alternatively the test script will interact with the FW via serial port.

This setup reduces the need to update the agent every time we improve the test script. The test script will be obtained from the branch being tested, allowing us to test testing scripts in PRs as well.

test/cc3200

This build environment is like test:esp-12e but comes with a cc3200 device and flashing tool installed.

Workflow execution and data model

Until now, we focused on agents and the environment they have to provide in order to execute individual build tasks. However, a build job is composed by a bunch of build tasks which are orchestrated in order to achieve a single goal.

Unlike some workflow tools, build tasks are not defined statically, but instead are spawned by other build tasks.

State

This makes it easy to run builds and delegate build logic to specific build scripts, but how can we gather the results of the build steps?

Each time a build task is spawned it’s associated with a build job. The build job will keep track of each outstanding build task and its completion status. A build job is completed only when all its tasks are completed.

An agent can create tasks and change in task status (start execution, completion, etc) by calling a clubby method on the OurCI master.

Consistency

All task creation is done via OurCI master, which keeps in memory state and restarts all builds for not green pull requests (like it does now). Persistency can be added later.

UI

The new UI will be built on top of the cloud/dashboard UI.

The UI will communicate with the OurCI master via JS clubby.

During initial page load the UI will obtain the initial list of build jobs with a clubby call to the OurCI master. The UI will subscribe to Pub Sub notifications to keep the UI up to date, and use the Log Store API to stream the currently selected build logs.

The UI will render a list of build jobs. When clicking on a build job, it will show a flat list of currently scheduled build tasks. No attempt is made to show a tree or a graph of the workflow.

When clicking on a build task, it will show the log for that build task.

Other execution modes

The granularity and independence of our build tasks makes it possible to exploit some build agents independently from OurCI main workflow. We can spawn build tasks from external CI services like CircleCI, provided that the external CI build script can perform a clubby call (perhaps via REST API). To better suit external use cases, we can tailor HW agents for that use-case by using a separate topic that implements a different contract in its build environment (e.g. a more external-user friendly way to get build artefacts).

Agents

Agents are simple clubby clients that can spawn a subprocess, capture it’s output and stream it back via clubby. The easiest way to implement it currently is in Go, since we can take the log tailer code out of the existing OurCI. It’s possible to write it using MGIOT Posix as well but at the moment it would be a waste of time to duplicate code and depend on MGIOT to be bug free; after all this is a continuous integration system; it should be more stable than what it's testing.

Agents don’t have to be registered. All they need to do is to poll to a subscription. Here is the diagram taken from the Pub Sub design doc that illustrates the difference between a topic a subscription and a subscriber:

Design_Doc_OurCI_2.png

In order for a bunch of agents to share the same task queue, each agent will pull messages from a subscription. The subscription has a well-known name. For this use case it can have the same name as the topic itself.

Since agents pull for tasks, tasks will be consumed as soon as agents are ready. The more agents pull from the same queue, the faster we’ll progress. Agents can come and go dynamically. If an agent disappears without acknowledging or replying to a Pub Sub message, the Pub Sub service will make the message available again and another agent will pick it. Messages will have an ETA and will be dropped once expired. The rest of the details are deferred to the document describing the Pub Sub service.

GCE (Google Cloud Engine) instance group autoscaler can be configured to add more instances if the load is to high. We can implement better policies by monitoring the Pub Sub queue length.

Example

This is just an example of how we can spawn build tasks that execute builds (each of a subset of our code base). Some tasks spawn other tasks. The reason might be grouping (keeping all Mongoose IoT Platform build logic together instead of scattering it into a big master rules file), or sequencing (waiting for build artefacts to be built before spawning a test task).

This example shows how a simple distributed build can be orchestrated with those primitives.

Workflow example

(All our design docs are collaborative in their nature, so this is an internal explanation to the team):This section started off as an attempt by Sergey to summarise what’s going on. But, it grew organically to address requests for clarifications. it contains duplicate information; sorry for the mess -- Marko

  1. Someone sends Mongoose PR
  2. OurCI master process, running on ci.cesanta.com, catches GitHub notifications and sends a message containing the “all.sh” command and a name of a docker container to the “ourci/build/docker” topic. This message is depicted in the leftmost green box in the figure 1.
  3. An agent running on a GCE machine having access to docker and srcfs is subscribed to the “ourci/build/docker” topic. It runs the command contained in the message invoking the “all.sh” script. The script then sends more messages to the same “ourci/build/docker” topic thus spawning V7, Mongoose Embedded Web Server and Mongoose IoT Platform builds. The second column of green boxes depicts this wave of messages.
  4. Respective agents that perform builds for V7, Mongoose Embedded Web Server and Mongoose IoT Platform, will execute each their own shell script within a docker container specified in the published message defining the task. The first wave of messages will be executed in our basic build environment (the one that currently executes Makefile.ci).
  5. The Mongoose IoT Platform build script spawns a few child tasks. Figure 1 depicts only the spawn of esp8266 crosstool build environments (the script detail below shows how it also spawns a Posix build in parallel, although it technically can run in the context of this build task as well).
  6. The esp8266 cross builds will still be executed by the agent running on GCE (since it’s sent to the ourci/build/docker topic) but this time it will use the esp SDK docker image instead of the generic build base image.
  7. When the Mongoose IoT Platform build tasks produce a firmware successfully, the script will upload a FW tarball and a test script to the blobstore and publish another build task request message to the ourci/test/esp-12e topic.
  8. The agents running on shepetovka class machines will then pick up this task, download the script and run it. The script will download the FW, flash it on the device and perform other steps that are out of the scope of this document but can be briefly summarised as: use cadaver to upload a JS test script to the device, reboot it, wait for the device to report test results by listening to a socket (e.g. with netcat or perhaps Posix Mongoose IoT Platform itself downloaded from the blobstore)
  9. The end state is reached when all currently open tasks are being closed. The OurCI master then can treat the job as closed and set the GitHub PR status according to whether all tasks have succeeded or not. OurCI will set the PR status as failed as soon as the first task fails. In our first iteration there are no plans to abort already spawned tasks. Once they fail they will contribute to the still open build job, however, the PR will immediately reflect the failed status (and notification emails will be send to author and possibly reviewer).
  10. If we leave all the garbage in the blobstore we would accumulate an estimated of 7 GB of FW tarballs per architecture per year. The content of the blobstore can be cleaned up with a cron job, thus, allowing us to easily fetch the same images and try them out interactively or use them along with core dumps to get stacktraces. I don’t see another reason to not clean the Blob Store when the build job is marked as done (either because it’s closed because all task have reported completion or because OurCI master closed it forcibly because of timeout).

Agent A, running on the CI machines on GCE:

$ ourci_agent -t ourci/build/docker ourci_docker_agent.sh

The master script being run by the agent on each build is just a trampoline to a script stored in the repo:

$ cat ourci_docker_agent.sh
#!/bin/bash
BRANCH=$1

TMPCLONE=$(mktemp -d)
git clone --reference /data/srcfs/dev -b "${BRANCH}" \   
         git@github.com:cesanta/dev.git "${TMPCLONE}"

function cleanup {
   rm -rf ${TMPCLONE}
}
trap cleanup EXIT

# see current ourci/cmd/ourciweb/tools/ourci
function run() {
...

cd ${TMPCLONE}

IMAGE="$1"
SCRIPT="$2"
shift

shift
run ${SCRIPT} "$@"


The actual build logic is stored in bash scripts, makefiles whatever. Here’s a simple example where the build is defined with a set of build scripts that spawn other build tasks whose bodies are themselves defined in shell scripts:

$ cat /data/srcfs/dev/ourci/scripts/functions.sh
...
ESP_SDK=$(cat smartjs/platforms/esp8266/sdk.version)
BASE_SDK=docker.cesanta.com:5000/ourci

# publish a build task request to the "-t <topic>"
# encoding the rest of the arguments in the message

# the newly created task will be linked to the current job
# held in the $BUILD_JOB env
function spawn_task() {
...

OurCI master will spawn the first task and pass “all.sh” as parameter.

$ cat /data/srcfs/dev/ci/all.sh
#/bin/bash
. $(dirname $0)/ourci/scripts/functions.sh

spawn_task -t ourci/build/docker ${BASE_SDK} "ci/v7.sh"
spawn_task -t ourci/build/docker ${BASE_SDK} "ci/mongoose.sh"
spawn_task -t ourci/build/docker ${BASE_SDK} "ci/cloud.sh"
spawn_task -t ourci/build/docker ${BASE_SDK} "ci/fw.sh"

Simple builds can be done by a shell script or even a makefile

$ cat /data/srcfs/dev/ci/v7.sh
#/bin/bash
. $(dirname $0)/ourci/scripts/functions.sh

# we can further split the build steps in smaller pieces to be run in parallel
# or do something more clever in future
$(MAKE) -C v7/tests compile

$(MAKE) -C v7 w build_variants -j8
$(MAKE) -C v7/docs
$(MAKE) -C v7 v7.c difftest
$(MAKE) -C v7 run difftest
$(MAKE) -C v7 run CFLAGS_EXTRA=-m32
$(MAKE) -C v7/examples
git checkout v7/tests/ecma_report.txt

if [ "x$(CI_BRANCH)" = xmaster ]; then $(MAKE) -C v7 report_footprint; fi

Bigger components such as Mongoose IoT Platform can spawn several sub tasks, some of which might need to run in a different container:

$ cat /data/srcfs/dev/ci/fw.sh
#!/bin/bash
. $(dirname $0)/ourci/scripts/functions.sh

spawn_task -t ourci/build/docker ${BASE_SDK} "ci/fw/posix.sh"
spawn_task -t ourci/build/docker ${ESP_SDK} "ci/fw/esp8266.sh"
spawn_task -t ourci/build/docker ${ESP_SDK} "OTA=1 ci/fw/esp8266.sh"
...

A HW test is spawned after a build is made within the cross toolchain image:

$ cat /data/srcfs/dev/ci/fw/esp8266.sh
#!/bin/bash
. $(dirname $0)/ourci/scripts/functions.sh

SCRIPT_BLOB=/ourci/scripts/esp-12e/${SHA}
FW_BLOB=/ourci/fw/esp-12e/${SHA}.tar.gz

make -C fw/platforms/esp8266 -f Makefile.build OTA=${OTA}
tar cvfz fw.tar.gz -C fw/platforms/esp8266 firmware

blob_upload fw.tar.gz ${FW_BLOB}
blob_upload ci.fw.esp8266_test_driver.sh ${SCRIPT_BLOB}

spawn_task -t ourci/test/esp-12e ${FW_BLOB} ${SCRIPT_BLOB}

The actual test driver is stored in the repo and copied into the blobstore so the HW test agent can run it without needing git credentials:

$ cat /data/srcfs/dev/ci/fw/esp8266_test_driver.sh
#!/bin/bash
curl ${BLOB_URL}/${1} >fw.tar.gz
tar ...
esptool.sh ...
sleep ...
cadaver ...
sleep ...
curl ...

Agent B, running on shepetovka.corp.cesanta.com:

Agent of shepetovka (this is the hostname of an internal test machine) has a very simple script. The actual per-build script will be fetched from the blobstore on each build:

$ ourci_agent -t ourci/test/esp-12e ourci_esp8266_agent.sh
$ cat ourci_esp8266_agent.sh
#!/bin/sh

curl ${BLOB_URL}/${2} >test_script.sh
chmod +x test_script.sh
./test_script ${1}

Caveats

The price paid for keeping the workflow model dead simple, non-declarative and without explicit dependencies, is that we have to be explicit in our sequencing. If we need to spawn a task after some task did something, we have to put the spawn command after that something, which means that the “spawn ESP test” command will be buried in some non-top level script.

If this proves to be too cumbersome we can improve the way we describe the build.

Join Mongoose IoT Cloud

Comments