Scheduling and Orchestration in Quant Research Infrastructure

A dependency-aware scheduler, run instead of a fixed cron job, keeps a pipeline honest about whether its data is ready.

Scheduling and orchestration in quant research infrastructure means running a defined sequence of data pipeline steps, such as ingest, normalize, validate and store, on a repeatable schedule with explicit dependencies between steps, rather than relying on a person to run scripts by hand in the correct order each morning. The practical requirement that separates research orchestration from a general data engineering setup is that a single day's data has to be fully validated and available before any backtest or model run that depends on it starts, which means the scheduler has to track dependencies at the level of a specific date and symbol rather than only at the level of a job finishing.

A cron job runs a script at a fixed time regardless of whether the input it depends on is actually ready, which works acceptably for a single, independent task but breaks down as soon as one step's output feeds into another step's input, because a delayed upstream feed file or a slow vendor delivery leaves a downstream job either failing outright or, worse, running successfully against stale or partial data. A researcher who wakes up to a completed backtest has no way to tell from a cron log alone whether the input data behind that backtest was actually complete for the day.

The fix is to schedule based on the completion of an upstream step rather than a fixed wall-clock time, so that a normalization job triggers only once ingestion has actually finished successfully for that date, and a validation job triggers only once normalization has finished, which is the basic idea behind every orchestration tool built for this purpose.

Orchestration tools such as Airflow, Prefect and Dagster represent a pipeline as a directed acyclic graph, where each node is a task and each edge is a dependency, and the scheduler only runs a task once every task it depends on has completed successfully. This representation makes the dependency structure explicit and inspectable rather than implicit in the order scripts happen to be listed in a shell script, which matters once a pipeline grows past a handful of steps and several people are maintaining it.

The graph structure also makes partial reruns possible: if only the validation step for a single date failed, the graph shows exactly which downstream tasks depended on that step and need to rerun, without forcing a full rebuild of every upstream step that already succeeded.