Skip to main content

Parallelism

This page will discuss how to run multiple tasks in parallel for a single Job.

Motivation

A single Job without parallelism will have at most one task running at any time, which corresponds to a single Pod (if the pod task executor is used).

Assume we have a JobConfig for processing all rows in a database, and is scheduled to run every day, which creates a Job (and thus a Pod) that will run to completion every day. Over time as the database grows in size, the Job may take longer to run since it now has more items to process, and may eventually take too long for business requirements. Apart from vertically scaling the Pod, you may wish to parallelize the job processing into multiple partitions, which can be achieved by specifying the number of partitions via withCount.

Other means of expanding a job into multiple parallel tasks include one parallel task for each item in a list (withKeys), or for each combination of values in a matrix (withMatrix).

The equivalent feature in Kubernetes is Indexed Jobs.

Sample Configuration

apiVersion: execution.furiko.io/v1alpha1
kind: Job
spec:
parallelism:
withCount: 3
completionStrategy: AllSuccessful
template:
taskTemplate:
pod:
spec:
containers:
- name: job-container
image: my-image
args:
- ./bin/consume -shard-index=${task.index_num}

Configuration Options

Exactly one of withCount, withKeys, or withMatrix must be specified.

withCount

Specifies the number of parallel tasks to be executed in parallel. Must be a positive integer.

If withCount is used, then the task.index_num context variable will be available to substitute the index number (from 0 to N-1) as a context variable.

withKeys

Specifies keys that will correspond to each parallel task.

If withKeys is used, then the task.index_key context variable will be available to substitute the index key as a context variable.

Each key must be a non-empty string.

withMatrix

Specifies a matrix of key-value pairs, such that each matrix combination will correspond to each parallel task. For example, given the following configuration:

withMatrix:
goos:
- linux
- darwin
- windows
goarch:
- amd64
- arm64

A total of 6 tasks will be run in parallel:

Taskgoosgoarch
0linuxamd64
1linuxarm64
2darwinamd64
3darwinarm64
4windowsamd64
5windowsarm64

If withMatrix is used, then the task.index_matrix.<key> context variable will be available to substitute the index key as a context variable.

Each key must be a non-empty string and contain only lowercase letters, numbers, underscores, and dashes. Each value must be a non-empty string.

completionStrategy

Determines when the Job is complete when multiple tasks are running in parallel.

Must be one of the following values.

NameDescription
AllSuccessfulCompleted only once each parallel index has succeeded.
AnySuccessfulCompleted when any parallel index has succeeeded.

AllSuccessful

This strategy means that the Job is completed only once each parallel index has succeeded, and the completion strategy is deemed to have succeeded.

If any parallel index has exhausted all its retries (i.e. it has failed), then the completion strategy is deemed to have failed. All remaining tasks will be terminated.

AnySuccessful

This strategy means that the Job is completed when any parallel index has succeeeded, and the completion strategy is deemed to have succeeded. All remaining tasks will be terminated.

If all parallel indexes have exhausted all of their retries (i.e. it has failed), only then will the completion strategy be deemed to have failed.