HyperMake is a parameterized pipeline definition language (think of a make where tasks can be parameterized) heavily inspired by Ducttape.
- Shell scripting: Write tasks in plain Bash, just like
make. No Python or YAML for defining tasks. - Cached intermediate results: Intermediate results are cached, so that if a task fails, it can be re-run from the last successful task.
- Parameterization of tasks: a task can be of multiple versions (e.g. in a ML pipeline, a task can be of different hyperparameters).
- Minimal juggling: Inputs and outputs are just files/symlinks, and arguments are all passed as environment variables. Tasks are realized as a child process.
- Automatic parallelization: based on the dependency DAG.
- Cloud-agnostic: tasks can be run locally, or on a cloud (e.g. AWS, Azure), with minimal changes to the pipeline.