Distributed job scheduler

Elect one leader with a time-limited lease (etcd, ZooKeeper, or a Redis lock with a fencing token) to claim due jobs. Or let every worker claim rows with SELECT … FOR UPDATE SKIP LOCKED, which gives each due job to exactly one claimer. Either way, make the job idempotent, because a crash after the work but before the acknowledgement will cause a retry.

Clients register jobs (one-off or cron) through an API that stores definitions and next run times in a database. A scheduler component, with a single active leader or partitioned ownership, finds due jobs and enqueues them. Worker pools pull from queues, execute with timeouts, report status and heartbeats, and failed runs are retried or sent to a dead-letter queue.

Index jobs by next_run_at and poll in small batches (WHERE next_run_at <= now() ORDER BY next_run_at LIMIT n FOR UPDATE SKIP LOCKED), or use a timing wheel or Redis sorted set scored by run time. After claiming, compute and store the next run time for recurring jobs.

Jobs are only marked done after the worker reports success; a lease with a timeout returns stuck jobs to the queue. Because a job might run twice, make handlers idempotent with job run IDs and idempotency keys for external effects.

Retry with exponential backoff and jitter, cap the number of attempts, and move the job to a dead-letter queue after the last one. Separate retryable errors (timeouts) from permanent ones (bad input), which should go straight to the DLQ.

When a worker claims a job, it gets exclusive ownership for a period; it must finish or extend the lease with heartbeats. If the worker dies, the lease expires and another worker can pick the job up.

Break them into smaller steps with checkpoints, send heartbeats to extend leases, store progress so retries resume rather than restart, and give them a separate queue and worker pool so they do not block short jobs.

Spread load with jitter where exact time is not required, partition jobs across scheduler shards, enqueue in batches ahead of time, and scale workers predictively before known peaks.

Use separate queues per priority with weighted consumption, or a priority field in the queue, and reserve capacity for high-priority jobs so a flood of low-priority work cannot starve them.

Store each schedule with its IANA time zone, compute next run times with a proper library, and define behaviour for skipped or repeated local times during DST changes, documenting it for users.

Store jobs as nodes with edges, run a job only when all its parents succeed, track run state per DAG execution, and support retries per node; tools like Airflow, Temporal or Step Functions implement this.

Record each run with state (queued, running, succeeded, failed, cancelled), timestamps, attempts, logs and output references, and provide an API and UI to list runs, retry, cancel or pause schedules.

Combine at-least-once delivery with an idempotent handler: give each run a unique run ID, record completion in the same transaction as the job's side effect or check a dedupe table before acting, and use idempotency keys when calling external APIs so a retry after a crash does nothing new.

Store jobs with a due time in a table indexed by due time, or in a Redis sorted set scored by timestamp. A poller moves due jobs into the ready queue in small batches with SKIP LOCKED or an atomic pop, so millions of future jobs cost nothing until they are due.

It is a number that increases with every new lease. If an old leader pauses (say, a long GC pause) and wakes up after its lease expired, its writes carry an old token and the storage rejects them, so two leaders cannot both act.

Mark the run cancelled in the store; workers check a cancellation flag periodically or receive a signal, stop work at a safe point and clean up. Jobs that cannot stop safely should finish and ignore their results.

A scheduler triggers jobs at times or intervals. A workflow engine (Temporal, Step Functions) orchestrates multi-step processes with state, retries, waits and compensation, often triggered by a scheduler.

Apply concurrency limits per job type or tenant, rate limits for external APIs, and backpressure from queue depth; spread retries with jitter.

Run several scheduler instances with leader election (etcd, ZooKeeper or database locks with fencing tokens) or partition jobs so each instance owns a shard; workers are stateless and scale horizontally.

Retry transient failures with exponential backoff and jitter up to a maximum, classify permanent errors to fail fast, and move exhausted jobs to a dead-letter queue with alerts.

Track usage per tenant, apply per-tenant concurrency and rate limits, and use fair queuing so one tenant's burst does not delay everyone else's jobs.

Use a fake clock to test due-time calculation and DST edge cases, simulate worker crashes to verify lease expiry and retries, and load-test bursts of due jobs.

Stream logs to a central logging system tagged with job and run IDs, store large outputs in object storage, and keep links in the run record with retention policies.

Classify the error as permanent, stop retrying, move the job to a dead-letter queue with its payload, error and attempt history, alert the owning team, and provide a tool to fix and replay it once the input or code is corrected.

Store a status on the schedule definition, skip creating new runs while paused, decide whether missed runs should be skipped or caught up on resume, and show the next run time so the behaviour is clear in the UI.

Split it into chunks, enqueue a child job per chunk, track progress in a parent record, cap concurrency so the database is not overwhelmed, make each chunk idempotent, and finish with a completion job once all children succeed.

Elect one leader with a lease in a coordination store, or let every instance try to insert the run with a unique key on schedule ID plus scheduled time so only one insert succeeds and the rest are ignored.

Cloud schedulers (EventBridge Scheduler, Cloud Scheduler), queue-based libraries (BullMQ, Sidekiq, Celery), workflow engines (Temporal, Airflow, Inngest), and Kubernetes CronJobs, depending on scale and needs.

Schedule lag (actual start minus due time), queue depth, run duration, success and failure rates, retries, dead-letter counts and worker utilisation.

Decide a misfire policy per job: run once immediately, run every missed occurrence, or skip to the next schedule, and record the choice.

Keep job payloads backward compatible, version handlers, drain old versions before removing them, and avoid changing the meaning of queued jobs.

Multiply jobs per second by average runtime to get the concurrent jobs needed, add headroom for peaks and retries, check per-worker memory and CPU, and auto-scale workers on queue depth and job age rather than CPU alone.

1. AEach claim different rows without waiting on each otherCorrect answer
2. BDelete locked rows
3. CBypass indexes
4. DRead uncommitted data

Why: Rows another transaction has locked are skipped, so workers grab different jobs in parallel.

1. ADeleting the job
2. BA new cron entry
3. CLease or visibility timeout expiryCorrect answer
4. DManual restart only

Why: Another worker can then claim it.

1. AIt is required by cron
2. BAt-least-once delivery may run a job twiceCorrect answer
3. CTo run faster
4. DTo save logs

Why: Duplicates must be harmless.

1. ATRUNCATE jobs
2. BLOCK TABLE jobs
3. CSELECT … FOR UPDATE SKIP LOCKEDCorrect answer
4. DSELECT * without locks

Why: Locked rows are skipped.

1. ATo make retries faster
2. BTo stop many clients retrying at the same instantCorrect answer
3. CTo encrypt payloads
4. DTo reduce log size

Why: Randomness spreads retries out, avoiding a thundering herd against the recovering service.

1. ALarger servers
2. BDisabling timeouts
3. CRunning twice
4. DHeartbeats that extend the leaseCorrect answer

Why: No heartbeat means the lease expires.

1. AUse UTC offsets only forever
2. BStore IANA time zones and a defined policyCorrect answer
3. CUse server local time
4. DIgnore DST

Why: Skipped or repeated hours need rules.

1. ALonger timeouts
2. BRandom ordering
3. CSeparate queues with reserved capacityCorrect answer
4. DOne FIFO queue

Why: This isolates priorities from each other.

1. Acrontab
2. Brsync
3. Ccurl
4. DTemporalCorrect answer

Why: Workflow engines handle retries and waits.

1. AA log format
2. BA priority level
3. CWhat to do about runs missed while downCorrect answer
4. DA retry limit

Why: Run once, run all or skip.

1. AInto logs only
2. BA dead-letter queue with alertsCorrect answer
3. CBack to the front of the queue forever
4. DNowhere

Why: Investigate and replay later.

1. AAt-least-once delivery plus idempotent handlersCorrect answer
2. BExactly-once network delivery
3. CLonger timeouts
4. DBigger workers

Why: Duplicates are expected; the handler makes them harmless.

1. AA sorted set scored by due timeCorrect answer
2. BA plain list
3. CA string per minute
4. DA pub/sub channel

Why: Due jobs are fetched by score range cheaply.

1. AThe same job running once per serverCorrect answer
2. BAutomatic failover
3. CLower memory use
4. DFaster jobs

Why: Each server's cron fires independently, so a job meant to run once runs N times.

1. ASchedule lagCorrect answer
2. BJob code size
3. CNumber of tenants
4. DDisk free

Why: Actual start minus due time.

1. AJitter where exact time is not neededCorrect answer
2. BMore cron entries
3. CShorter timeouts
4. DFewer workers

Why: Jitter smooths the load.

1. ALarger payloads
2. BMore retries
3. CTurning off logging
4. DConcurrency and rate limitsCorrect answer

Why: Respect their capacity.

1. ASet a flag workers check and stop safelyCorrect answer
2. BKill the database
3. CDelete the queue
4. DRestart all workers

Why: Stop at a safe point.

1. AA single shared queue
2. BFirst come only
3. CRandom drops
4. DPer-tenant limits and fair queuingCorrect answer

Why: One burst should not delay all.

1. AA cache
2. BA CDN
3. CA load balancer
4. DA DAG executorCorrect answer

Why: Dependencies form a graph.

1. ALeader election or partitioned ownershipCorrect answer
2. BOne big server
3. CManual failover only
4. DNo database

Why: Avoid single points of failure.

1. ANetwork speed
2. BDisk size
3. CDue times and DST edge casesCorrect answer
4. DCSS layout

Why: Time logic is tricky.

1. ARun on another region
2. BMoved to a dead-letter queue after classificationCorrect answer
3. CRetried forever
4. DDeleted silently

Why: Permanent errors need a person, not more retries.

1. ADuplicate cron runs from several instancesCorrect answer
2. BSlow jobs
3. CLong queues
4. DTime-zone bugs

Why: Only one insert for each slot can succeed.

1. AA cron job every second
2. BIdempotent chunks tracked by a parentCorrect answer
3. COne giant transaction
4. DA single worker loop with no checkpoints

Why: Chunks can retry and scale independently.

1. ATime of day only
2. BQueue depth and job ageCorrect answer
3. CDisk size
4. DNumber of schedules defined

Why: These show work waiting, not just busy CPUs.

1. ARoute 53
2. BEBS
3. CEventBridge SchedulerCorrect answer
4. DCloudFront

Why: Managed cron and one-off schedules.

1. AThe scheduler's memory
2. BCookies
3. CObject storage linked from the run recordCorrect answer
4. DThe queue message

Why: Queue messages should stay small.

1. AOld jobs being processed wronglyCorrect answer
2. BFaster processing
3. CBetter logs
4. DNothing

Why: Keep payloads compatible.

1. AHidden
2. BIgnored
3. CFailed fast without retriesCorrect answer
4. DRetried forever

Why: Retries cannot fix them.

1. ACronJobsCorrect answer
2. BDaemonSets
3. CIngress
4. DConfigMaps

Why: Built-in cron for clusters.

1. AThe cron syntax
2. BWhether missed runs are skipped or caught upCorrect answer
3. CWhich database to use
4. DThe worker's language

Why: Both are valid; users need to know which applies.