Parallelism

This page will discuss how to run multiple tasks in parallel for a single Job.

Motivation

A single Job without parallelism will have at most one task running at any time, which corresponds to a single Pod (if the pod task executor is used).

Assume we have a JobConfig for processing all rows in a database, and is scheduled to run every day, which creates a Job (and thus a Pod) that will run to completion every day. Over time as the database grows in size, the Job may take longer to run since it now has more items to process, and may eventually take too long for business requirements. Apart from vertically scaling the Pod, you may wish to parallelize the job processing into multiple partitions, which can be achieved by specifying the number of partitions via withCount.

Other means of expanding a job into multiple parallel tasks include one parallel task for each item in a list (withKeys), or for each combination of values in a matrix (withMatrix).

The equivalent feature in Kubernetes is Indexed Jobs.

Sample Configuration

With Count

apiVersion: execution.furiko.io/v1alpha1
kind: Job
spec:
  parallelism:
    withCount: 3
    completionStrategy: AllSuccessful
  template:
    taskTemplate:
      pod:
        spec:
          containers:
            - name: job-container
              image: my-image
              args:
                - ./bin/consume -shard-index=${task.index_num}

With Keys

apiVersion: execution.furiko.io/v1alpha1
kind: Job
spec:
  parallelism:
    withKeys:
      - golang
      - python
      - nodejs
    completionStrategy: AllSuccessful
  template:
    taskTemplate:
      pod:
        spec:
          containers:
            - name: job-container
              image: my-image
              args:
                - ./bin/run -lang=${task.index_key}

With Matrix

apiVersion: execution.furiko.io/v1alpha1
kind: Job
spec:
  parallelism:
    withMatrix:
      goos:
        - linux
        - darwin
      goarch:
        - amd64
        - arm64
    completionStrategy: AllSuccessful
  template:
    taskTemplate:
      pod:
        spec:
          containers:
            - name: job-container
              image: my-image
              env:
                - name: GOOS
                  value: ${task.index_matrix.goos}
                - name: GOARCH
                  value: ${task.index_matrix.goarch}
              args:
                - go build ./...

Configuration Options

Exactly one of withCount, withKeys, or withMatrix must be specified.

withCount

Specifies the number of parallel tasks to be executed in parallel. Must be a positive integer.

If withCount is used, then the task.index_num context variable will be available to substitute the index number (from 0 to N-1) as a context variable.

withKeys

Specifies keys that will correspond to each parallel task.

If withKeys is used, then the task.index_key context variable will be available to substitute the index key as a context variable.

Each key must be a non-empty string.

withMatrix

Specifies a matrix of key-value pairs, such that each matrix combination will correspond to each parallel task. For example, given the following configuration:

withMatrix:
  goos:
    - linux
    - darwin
    - windows
  goarch:
    - amd64
    - arm64

A total of 6 tasks will be run in parallel:

Task	goos	goarch
0	linux	amd64
1	linux	arm64
2	darwin	amd64
3	darwin	arm64
4	windows	amd64
5	windows	arm64

If withMatrix is used, then the task.index_matrix.<key> context variable will be available to substitute the index key as a context variable.

Each key must be a non-empty string and contain only lowercase letters, numbers, underscores, and dashes. Each value must be a non-empty string.

completionStrategy

Determines when the Job is complete when multiple tasks are running in parallel.

Must be one of the following values.

Name	Description
AllSuccessful	Completed only once each parallel index has succeeded.
AnySuccessful	Completed when any parallel index has succeeeded.

AllSuccessful

This strategy means that the Job is completed only once each parallel index has succeeded, and the completion strategy is deemed to have succeeded.

If any parallel index has exhausted all its retries (i.e. it has failed), then the completion strategy is deemed to have failed. All remaining tasks will be terminated.

AnySuccessful

This strategy means that the Job is completed when any parallel index has succeeeded, and the completion strategy is deemed to have succeeded. All remaining tasks will be terminated.

If all parallel indexes have exhausted all of their retries (i.e. it has failed), only then will the completion strategy be deemed to have failed.