Skip to content

Pipeline DSL

Writing Jenkins pipelines typically involves a lot of repetitive boilerplate and Groovy magic that is irrelevant to both repository maintainers and most contributors. We abstract all of that into a shared library.

Our goals:

  • Pipeline code must be writable by repository maintainers and regular contributors without any prior Jenkins experience
  • Pipeline code must be readable by casual contributors
  • It should be possible to approximate pipeline execution in a local development environment without spinning up a full Jenkins instance (launch the same container as in CI, execute the same commands inside)

Quick demo

CI pipeline for every repository is defined in .ci/Jenkinsfile (stored inside the corresponding repo).

Here is a basic example:

async {
    task('check') {
        dockerfile('./Dockerfile') {
            stage('build') {
                sh 'make build'
            }
            stage('test') {
                sh 'make test'
            }
        }
    }
    task('lint') {
        container('cool-linter:v12.3.4') {
            sh 'make lint'
        }
    }
}

Demo pipeline visualized

The pipeline above will be executed for every incoming patchset:

  • "Check" and "lint" tasks will always be started simultaneously
    • Both tasks will be executed in their own environments. They can not affect each other and can not share files/artifacts.
    • Both tasks will checkout current pull request commit into working directory
    • If one of the tasks fails the remaining one will run until completion.
    • Any task failure will mark the whole pipeline as failed.
  • "Lint" task will be executed in a container started from "cool-linter:v12.3.4" image (fetched from Docker Hub and cached for reuse).
  • "Check" task will be executed in a container that will be built on demand from a Dockerfile stored in the repository. Image builds will be cached automatically, rebuilds will be very fast.
  • "Check" task is split into two stages. Stages have no effect on control flow and are used only for pipeline visualization (see image above).
  • Task execution stops at the first failure. In the example above if 'make build' fails, 'make test' is never executed - this is desirable for C/C++ projects where tests require build artifacts to be present but in other scenarios (Go, Javascript) we could decouple unit tests from build step by placing them into separate tasks.

DSL overview

TrueCloudLab DSL introduces two new primitives: task{} and async{}:

  • task{} is the basic pipeline unit that runs in its own execution environment
    • This environment is guaranteed to be clean at the beginning of the task{} block (separate CI runner, clean workspace) and will be automatically destroyed at the end of the block. There is no need to explicitly invoke any cleanup steps.
    • Repository containing the Jenkinsfile where pipeline is defined will be automatically checked out at current patchset commit and will become available as current working directory.
    • Caches for common artifact types (OCI images, PyPI, Go modules, pre-commit hooks) are available implicitly and require no configuration. Cache entries are shared across pipelines and are invalidated on schedule.
  • async{} block starts all contained tasks simultaneously and waits for all tasks to complete before the end of the block.
    • Tasks are executed independently until completion and are not short-circuited if any of the sibling tasks fail.
    • Failure of any task within async{} block will mark the block (and the whole pipeline) as failed. No pipeline instructions after the failed async{} block will get executed.
    • Launching multiple tasks from async{} block is similar to firing off multiple goroutines from a wait group and calling wg.Wait() afterwards.
    • Using multiple consecutive async{} blocks in the same pipeline is allowed. Tasks in the next async{} block will start executing after all tasks in the previous one finish successfully.

Default behavior of async and task blocks may be tuned via optional parameters (refer to linked source code if you need to specify non-default agent label, disable implicit checkout or define agent-less tasks).

async{} and task{} blocks may be arbitrarily nested. Deep nesting may make pipelines very hard to reason about and may introduce hard-to-debug bugs (variable capture, closure scope mismatch, etc). It is therefore recommended to keep your pipelines reasonably flat.

The rest of pipeline is written using Jenkins scripted pipeline DSL (stage and sh are defined there).

  • stage has no effect on control flow and is used only for pipeline visualization (see image above).
  • sh executes the provided script with default POSIX shell using -xe flags. Refer to upstream documentation for details on using another shell and flags, capturing stdout or exit code
  • More pipeline steps are provided by Jenkins and plugins

Intermixing pipeline steps between async{} and task{} definitions is not recommended. Even though it's technically possible (TrueCloudLab DSL will not forbid this) the resulting control flow will be next to impossible to deduce analytically. Instead keep your pipeline steps inside task{} blocks and use only async{}, task{} and plain non-Jenkins Groovy syntax on top level.

Reproducible pipeline environment

Even though it's possible to execute pipeline steps directly on CI runner with a task{} block, doing so is not recommended:

// NOT RECOMMENDED (UNDEFINED BEHAVIOR)
task('Execute directly on CI runner') {
    sh 'make test'
}

Instead you should specify OCI container in which to execute your steps:

task('Execute in worker containers') {
    container('golang:1.25') {
        sh 'make test'
    }
}

Or reference a Dockerfile for an ad-hoc OCI container:

task('Execute in ad-hoc OCI container') {
    // Dockerfile from repository
    dockerfile('./Dockerfile') {
        sh 'make build'
    }

    // Inline Dockerfile
    dockerfile('''
        FROM python:3.10
        RUN apt update && apt install -y pre-commit
    ''') {
        sh 'make test'
    }
}

Everything inside task{} block will be executed sequentially. Using different containers for different steps is also supported:

task('Do something difficult') {
    container('node:25') {
        sh 'make codegen'
    }
    container('golang:1.25') {
        sh 'make build'
    }
    dockerfile('./docs/Dockerfile') {
        sh 'make publish'
    }
}

Pipeline helpers

General purpose pipeline helpers will also be added to our Jenkins library. The intention is similar to GitHub Actions but unlike GHA we do not support importing pipeline code from arbitrary remote repositories.

Currently implemented helpers: