concurrency of a JobConfig defines the behavior when a single JobConfig has multiple concurrent Jobs.
For example, the
cron.expression may schedule another Job while the previous Job is active, or an ad-hoc Job may be created while another Job is also active. The
concurrency.policy defines what to do in such a scenario.
policy defines the behavior when a single JobConfig has multiple concurrent Jobs.
When specified in a JobConfig, this controls the behavior of automatically scheduled Jobs and when Jobs are created on an ad-hoc basis (e.g. via
kubectl create). It is also specified as part of a Job's
startPolicy, which inherits the policy from the JobConfig, but could be overridden if so desired.
|Places no restriction on the number of Jobs that can be started concurrently whatsoever.|
|Forbids multiple Jobs to be started concurrently, and will be dropped/rejected immediately.|
|Places the incoming Job into a queue to be started once all other Jobs have finished.|
Allow policy means that no restriction is imposed on the number of Jobs that can be started concurrently whatsoever.
If a single JobConfig has a misconfigured
cron.expression that runs too frequently, this could potentially quickly exhaust all resources in the cluster.
Forbid policy means that any incoming Jobs that exceed the maximum concurrency will be dropped/rejected immediately.
When scheduled automatically using a schedule, if there is another ongoing Job, the newly scheduled Job will be ignored and not be created.
Attempting to run an ad-hoc Job with another ongoing Job will also result in an error:
cannot create job: admission webhook "validation.webhook.jobs.execution.furiko.io" denied the request: Job.execution.furiko.io "forbid-example-rglxf" is invalid: spec.startPolicy.concurrencyPolicy: Forbidden: forbid-example currently has 1 active job(s), but concurrency policy forbids exceeding maximum concurrency of 1
Enqueue policy means that any incoming Jobs that exceed the maximum concurrency will be placed into a queue to be started once all other Jobs have finished.
Any jobs that are queued would be treated as
Queued, and requires the JobQueueController to process the Job queue. Jobs will be processed in order of their
creationTimestamp in FIFO order.
When used to run an ad-hoc Job in conjunction with
startAfter, this allows you to start a Job after a given time. If instead
Forbid was used and there happens to be another ongoing Job at the time it was meant to be started, the Job would instead be rejected.
Note that Furiko does not currently support a maximum Job queue length, which could result in a huge backlog of Jobs to be started.
maxConcurrency allows you to configure the maximum number of Jobs that can be running concurrently. Defaults to
The exact behavior of
maxConcurrency depends on the
|If there are currently |
|If there are currently |
|Not allowed to specify |
In practice, we recommend using
Forbid as a sane default for most jobs for the following reasons:
- Many automation jobs are not concurrent-safe, and risk performing duplicate operations or running into TOCTTOU bugs when run concurrently.
- In an offline processing scenario, even if the previously running Job runs longer than normal, it would still be able to process all work to be done that would have been picked up by the later scheduled Job anyway.
One pattern that is commonly adopted is to:
- Specify a
cron.expressionwhich defines the maximum duration between any two Jobs being executed.
- This roughly equates to the maximum duration for an offline processing queue latency.
Preventing Concurrent Jobs
The following scenarios explain what happens when there is another concurrent Job when using the
- Another Job is about to be scheduled by the CronController: Will be silently dropped and from scheduling.
- An ad-hoc Job is created (e.g. using
kubectl): Will be rejected by the validating admission webhook in most cases.
If the Job was created almost at the same time that the concurrent Job started, it may be possible for it to bypass the validation webhook. In this case, the JobQueueController will reject the Job with
AdmissionError like as follows:
$ kubectl describe furikojob jobconfig-sample-btzqx
Type Reason Age From Message
---- ------ ---- ---- -------
Warning AdmissionRefused 12s JobQueueController Cannot start new Job, jobconfig-sample has 1 active Jobs but concurrency policy is Forbid
Warning Failed 12s JobController Job failed with result: AdmissionError
While Furiko tries its best to prevent concurrent Jobs, it is not a replacement for a distributed lock in the application. For highly critical operations, it is still recommended for Job developers to use traditional distributed locks or database systems (e.g. jobs processing money or transactions).