Pipelines

Pipelines allow you to fully customize the execution logic in DevSpace, i.e. they define what happens when a users run commands such as devspace dev, devspace build, devspace deploy or devspace run-pipeline my-custom-pipeline

Pipelines are defined in POSIX shell syntax and a DevSpace pipeline reads as a regular POSIX script. However, DevSpace implements certain special commands that can be used inside the POSIX script to tell DevSpace when to build, deploy or start developing. For a complete function reference, please take a look below.

Execution Order

# Simple deployment that deploys an nginx pod
deployments:
  test:
    helm:
      values:
        containers:
        - image: nginx

pipelines:
  deploy: |-
    # You can use the --set, --set-string, --from and --from-file
    # arguments to dynamically change the config of the images,
    # deployments or dev configurations you want to use inside the
    # pipeline.

    # Exchange the image in our deployment
    if is_equal ${DEVSPACE_NAMESPACE} "test"; then
      create_deployments test --set "helm.values.containers[0].image=bitnami/nginx"
    fi

    # Create a new deployment based on another deployment and change the image.
    # This will copy over the values from the test deployment and then change the image
    create_deployments nginx --from test \
                             --set helm.values.containers[0].image=mysql

    # Create a new deployment from a file. ${DEVSPACE_TMPDIR} is always cleaned up
    # after each run
    echo """
    helm:
      values:
        containers:
        - image: nginx
    """ > ${DEVSPACE_TMPDIR}/my-deployment.yaml
    create_deployments nginx-from-file --from-file ${DEVSPACE_TMPDIR}/my-deployment.yaml

    # Force the container name to be a string (true) as DevSpace will otherwise convert
    # those automatically.
    create_deployments nginx-string-annotation --from test \
                                               --set-string "helm.values.containers[0].name=true"

Dynamic Config

pipelines:
  deploy: |-
    # Pipelines are a great tool to define your custom execution order of
    # building, deploying and starting dev configurations. This works for
    # all special commands such as: build_images, create_deployments, start_dev
    # run_dependencies and run_pipelines.

    # Run two pipelines in parallel
    run_pipelines other-pipeline-1 other-pipeline-2

    # Wait until all 4 deployments were deployed
    echo "All deployments are deployed"

  other-pipeline-1: |-
    # Deploys deployments deploy-a and deploy-b in parallel
    create_deployments deploy-a deploy-b
    echo "Deployment deploy-a and deploy-b are deployed"

  other-pipeline-2: |-
    # Deploys deployments deploy-c and then deploy-d
    create_deployments deploy-c
    echo "Deployment deploy-c is deployed"
    create_deployments deploy-d
    echo "Deployment deploy-d is deployed"


deployments:
  deploy-a: ...
  deploy-b: ...
  deploy-c: ...
  deploy-d: ...

Rerun Pipeline

pipelines:
  deploy: |-
    # This pipeline watches a secret and applies it as well as deploys the application
    # Start two pipelines in parallel
    run_pipelines watch-secret default-deploy

  watch-secret: |-
    # Rerun the pipeline as soon as the secret.yaml changes
    run_watch -p secret.yaml -- kubectl apply -n ${DEVSPACE_NAMESPACE} -f secret.yaml

  default-deploy: |-
    # Run regular deploy pipeline to start the application
    run_default_pipeline deploy

deployments: ...
dev: ...

Deploy / Sync / Open

pipelines:
  # Run with devspace run-pipeline custom-pipeline
  custom-pipeline: |-
    # Deploy simple nginx pod with the default DevSpace component chart
    create_deployments nginx --set helm.values.containers[0].image=nginx

    # Create a file we want to sync to the nginx pod
    echo "Hello World!" > ${DEVSPACE_TMPDIR}/index.html

    # Sync the file into the nginx pod via DevSpace sync
    start_dev nginx --set imageSelector=nginx \
                    --set "sync[0].path=${DEVSPACE_TMPDIR}:/usr/share/nginx/html" \
                    --set "sync[0].noWatch=true"

    # Print contents within the nginx pod
    exec_container --image-selector=nginx -- cat /usr/share/nginx/html/index.html

    # Start port-forwarding and open the url
    start_dev nginx --set imageSelector=nginx \
                    --set 'ports[0].port=8080:80' \
                    --set 'open[0].url=http://localhost:8080'

Custom Dockerfile Flag

images:
  my-image:
    dockerfile: ./Dockerfile

pipelines:
  # Executed on 'devspace deploy'
  deploy:
    flags:
    - name: dockerfile
      description: If enabled, will build the image with the given dockerfile
      type: string
    run: |-
        if ! is_empty $(get_flag dockerfile); then
            echo "Building the image with the overriden dockerfile $(get_flag dockerfile)"
            run_default_pipeline deploy --set "images.my-image.dockerfile=$(get_flag dockerfile)"
        else
            run_default_pipeline deploy
        fi

Using Pipelines

1. Define Pipelines

An example pipelines section could look like this in devspace.yaml:

devspace.yaml

version: v2beta1
pipelines:
  # Override the default pipeline for 'devspace dev'
  dev: |-
    run_dependency_pipelines --all    # 1. Deploy any projects this project needs (see "dependencies")
    create_deployments --all          # 2. Deploy Helm charts and manifests specfied as "deployments"
    start_dev app                     # 3. Start dev mode "app" (see "dev" section)
  deploy: |-
    run_dependency_pipelines --all                    # 1. Deploy any projects this project needs (see "dependencies")
    build_images --all -t $(git describe --always)    # 2. Build, tag (git commit hash) and push all images (see "images")
    create_deployments --all                          # 3. Deploy Helm charts and manifests specfied as "deployments"
  build:
    # Here we are using the long-form using `run:` instead of passing the script directly
    run: |-
      build_images --all -t $(git describe --always)
  custom: ...

Pipelines can:

• Call built-in functions such as build_images, create_deployments or start_dev
• Call functions defined in devspace.yaml or in any import
• Use bash-like syntax including if []; then ... fi, &&, || or ;
• Access environment variables and DevSpace variables

2. Run Pipelines

Pipelines are invoked via:

devspace run-pipeline [name]

Cross-Platform Execution

Write all pipeline scrips in bash fashion. DevSpace is using a library to make them cross-platform executable.

Default Pipelines

Internally DevSpace uses pipelines for the following commands that can be overriden according to your preferences. DevSpace provides default pipeline scripts for the following top-level commands:

• devspace dev
• devspace deploy
• devspace build
• devspace purge

dev

The dev pipeline is executed by running devspace dev and has the following default workflow:

run_dependencies --all
ensure_pull_secrets --all
build_images --all
create_deployments --all
start_dev --all

deploy

The deploy pipeline is executed by running devspace deploy and has the following default workflow:

run_dependencies --all
ensure_pull_secrets --all
build_images --all
create_deployments --all

build

The build pipeline is executed by running devspace build and has the following default workflow:

run_dependencies --all --pipeline build
build_images --all

purge

The purge pipeline is executed by running devspace purge and has the following default workflow:

stop_dev --all
purge_deployments --all
run_dependencies --all --pipeline purge

Custom Flags

To add custom flags to commands such as devspace dev or devspace run-pipeline my-custom-pipeline, you can specify the flags field for the respective pipeline.

The following example defines two flags for devspace dev:

• --logs / -l with type not specified (defaults to bool)
• --env / -e with type: stringArray

devspace.yaml

version: v2beta1
pipelines:
  dev:
    flags:
    - name: logs
      short: l
    - name: env
      short: e
      type: stringArray
    run: |-
      extraEnv=($(get_flag "env"))    # Retrieve the value of the `env` flag and store it in an array variable
      echo ${extraEnv[0]}             # Arrays are zero indexed

      TERMINAL_ENABLED=true
      if [ $(get_flag "logs") == "true" ]; then     # Test if --logs/-l flag is used or not
        TERMINAL_ENABLED=false                      # Disable terminal, so DevSpace falls back to log streaming
      fi

      start_dev app --set terminal.enabled=$TERMINAL_ENABLED

Built-In Functions

DevSpace provides a set of built-in functions. There are two types of functions:

1. Pipeline-Only Functions
2. Global Functions

Pipeline-Only Functions

Pipeline-only functions can only be used inside the scripts within the pipelines section of devspace.yaml.

Images

build_images [image-1][image-2] ... pipeline only

Builds all images passed as arguments in parallel

--tag / -t []string pipeline only

If enabled will override the default tags

--skip bool pipeline only

If enabled will skip building

--skip-push bool pipeline only

Skip pushing

--skip-push-on-local-kubernetes bool pipeline only

Skip pushing

--force-rebuild bool pipeline only

Skip pushing

--sequential bool pipeline only

Skip pushing

--max-concurrent int pipeline only

A pointer to an integer

--all bool pipeline only

Build all images

--except []string pipeline only

If used with --all, will exclude the following images

--set []string pipeline only

Set configuration

--set-string []string pipeline only

Set configuration as string

--from []string pipeline only

Reuse an existing configuration

--from-file []string pipeline only

Reuse an existing configuration from a file

ensure_pull_secrets [image-1][image-2] ... pipeline only

Creates pull secrets for all images passed as arguments

--set []string pipeline only

Set configuration

--set-string []string pipeline only

Set configuration as string

--from []string pipeline only

Reuse an existing configuration

--from-file []string pipeline only

Reuse an existing configuration from a file

--all bool pipeline only

Ensure all pull secrets

--except []string pipeline only

If used with --all, will exclude the following pull secrets

get_image [image] string pipeline only

Returns the most recently built image and/or tag for a given image name

--dependency string pipeline only

Retrieves the image from the named dependency

--only string pipeline only

Displays either only the tag or only the image

Deployments

create_deployments [deployment-1][deployment-2] ... pipeline only

Creates all deployments passed as arguments in parallel

--skip-deploy bool pipeline only

If enabled, will skip deploying

--force-redeploy bool pipeline only

Forces redeployment

--sequential bool pipeline only

Sequentially deploys the deployments

--render bool pipeline only

If true, prints the rendered manifests to the stdout instead of deploying them

--set []string pipeline only

Set configuration

--set-string []string pipeline only

Set configuration as string

--from []string pipeline only

Reuse an existing configuration

--from-file []string pipeline only

Reuse an existing configuration from a file

--all bool pipeline only

Deploy all deployments

--except []string pipeline only

If used with --all, will exclude the following deployments

purge_deployments [deployment-1][deployment-2] ... pipeline only

Purges all deployments passed as arguments

--force-purge bool pipeline only

Forces purging of deployments even though they might be still in use by other DevSpace projects

--all bool pipeline only

Deploy all deployments

--except []string pipeline only

If used with --all, will exclude the following deployments

--sequential bool pipeline only

Sequentially purges the deployments

Dev

start_dev [dev-1][dev-2] ... pipeline only

Starts all dev modes passed as arguments

--continue-on-terminal-exit bool pipeline only

Continue on terminal exit

--disable-sync bool pipeline only

If enabled will not start any sync configuration

--disable-port-forwarding bool pipeline only

If enabled will not start any port forwarding configuration

--disable-pod-replace bool pipeline only

If enabled will not replace any pods

--disable-open bool pipeline only

If enabled will not auto-open the URL

--set []string pipeline only

Set configuration

--set-string []string pipeline only

Set configuration as string

--from []string pipeline only

Reuse an existing configuration

--from-file []string pipeline only

Reuse an existing configuration from a file

--all bool pipeline only

Start all dev configurations

--except []string pipeline only

If used with --all, will exclude the following dev configs

stop_dev [dev-1][dev-2] ... pipeline only

Stops all dev modes passed as arguments

--force-purge bool pipeline only

Forces purging of deployments even though they might be still in use by other DevSpace projects

--all bool pipeline only

Stop all dev configurations

--except []string pipeline only

If used with --all, will exclude the following dev configs

Pipelines

run_pipelines [pipeline-1][pipeline-2] ... pipeline only

Runs all pipelines passed as arguments

--background bool pipeline only

Run the pipeline in the background

--sequential bool pipeline only

Run pipelines one after another

--set-flag []string pipeline only

Set a pipeline flag

--set []string pipeline only

Set configuration

--set-string []string pipeline only

Set configuration as string

run_default_pipeline [pipeline] pipeline only

Runs the default pipeline passed as arguments

run_dependency_pipelines [dependency-1][dependency-2] ... pipeline only

Runs a pipeline of each dependency passed as arguments

--pipeline string pipeline only

The pipeline to deploy from the dependency

--exclude []string pipeline only

Dependencies to exclude

--only []string pipeline only

Dependencies to include

--sequential bool pipeline only

Run dependencies one after another

--set-flag []string pipeline only

Set a pipeline flag

--all bool pipeline only

Deploy all dependencies

--except []string pipeline only

If used with --all, will exclude the following dependencies

Checks

is_dependency int pipeline only

Returns exit code 0 if the pipeline currently being executed is run because the project is a dependency of another project

Other

select_pod string pipeline only

Returns the name of a Kubernetes pod

--image-selector string pipeline only

The image selector to use to select the container

--label-selector string pipeline only

The label selector to use to select the container

--container string pipeline only

The container to use

--namespace / -n string pipeline only

The namespace to use

--disable-wait bool pipeline only

If true, will not wait for the container to become ready

--timeout int64 pipeline only

The timeout to wait. Defaults to 5 minutes

wait_pod [command] pipeline only

Waits for a pod to become running

--image-selector string pipeline only

The image selector to use to select the container

--label-selector string pipeline only

The label selector to use to select the container

--container string pipeline only

The container to use

--namespace / -n string pipeline only

The namespace to use

--disable-wait bool pipeline only

If true, will not wait for the container to become ready

--timeout int64 pipeline only

The timeout to wait. Defaults to 5 minutes

exec_container [command] pipeline only

Executes the command provided as argument inside a container

--image-selector string pipeline only

The image selector to use to select the container

--label-selector string pipeline only

The label selector to use to select the container

--container string pipeline only

The container to use

--namespace / -n string pipeline only

The namespace to use

--disable-wait bool pipeline only

If true, will not wait for the container to become ready

--timeout int64 pipeline only

The timeout to wait. Defaults to 5 minutes

get_config_value [json.path] string pipeline only

Returns the value of the config loaded from devspace.yaml

Global Functions

Global functions can be used anywhere in devspace.yaml, either in config fields that expect a bash script such as within functions or using $(command) vars in any other config field.

Checks

is_empty [value] int pipeline only

Returns exit code 0 if the value of the argument is empty string

is_equal [value-1][value-2] int pipeline only

Returns exit code 0 if the values of both arguments provided are equal

is_in [value-1][value-2] int pipeline only

Returns exit code 0 if the value of the first argument can be found in the second argument (second argument being a blank-separated list of strings e.g "bananas apples peaches")

is_os [os] darwin linux windows aix android dragonfly freebsd hurd illumos ios js nacl netbsd openbsd plan9 solaris zos int pipeline only

Returns exit code 0 if the current operating system equals the value provided as argument

is_true [value] int pipeline only

Returns exit code 0 if the value of the argument is "true"

Other

cat [file-path] string pipeline only

Returns the content of a file

get_flag [flag-name] string pipeline only

Returns the value of the flag that is provided as argument

run_watch [command] pipeline only

Executes the command provided as argument and watches for conditions to restart the command

--fail-on-error bool pipeline only

If true the command will fail on an error while running the sub command

--skip-initial bool pipeline only

If true will not execute the command immediately.

--silent bool pipeline only

If true will not print any warning about restarting the command.

--skip-and-silent / -s bool pipeline only

If enabled will not print when the command is restarted and not execute the command initially.

--exclude / -e []string pipeline only

The paths to ignore. Can be patterns in the form of ./**/my-file.txt

--path / -p []string pipeline only

The paths to watch. Can be patterns in the form of ./**/my-file.txt

sleep [seconds] pipeline only

Pauses the script execution for the number of seconds provided as argument

xargs [command] pipeline only

Reads from stdin, splits input by blanks and executes the command provided as argument for each blank-separated input value (often used in pipes, e.g. echo 'image-1 image-2' | xargs build_images)

Config Reference

pipelines required <pipeline_name>:object

Pipelines are the work blocks that DevSpace should execute when devspace dev, devspace build, devspace deploy or devspace purge is called. Pipelines are defined through a special POSIX script that allows you to use special commands such as create_deployments, start_dev, build_images etc. to signal DevSpace you want to execute a specific functionality. The pipelines dev, build, deploy and purge are special and will override the default functionality of the respective command if defined. All other pipelines can be either run via the devspace run-pipeline command or used within another pipeline through run_pipelines.

<pipeline_name> required string dev
deploy
build
purge
.*

Name of the pipeline, will be filled automatically

run required string

Run is the actual shell command that should be executed during this pipeline

flags required object[]

Flags are extra flags that can be used for running the pipeline via devspace run-pipeline.

name required string

Name is the name of the flag

short required string

Short is optional and is the shorthand name for this flag. E.g. 'g' converts to '-g'

type required string bool bool
int
string
stringArray

Type is the type of the flag. Defaults to bool

default required

Default is the default value for this flag

description required string

Description is the description as shown in devspace run-pipeline my-pipe -h

continueOnError required boolean false

ContinueOnError will not fail the whole job and pipeline if a call within the step fails.