Apache Hamilton

Hamilton is a framework for writing data pipelines, ML features, and LLM workflows as ordinary Python functions. You write small functions with typed inputs and outputs; Hamilton stitches them into a directed acyclic graph, handles execution order, caching, and parallelization, and gives you lineage and visualization for free.

The appeal is that there's no new DSL to learn and no YAML orchestration layer: your pipeline is just functions, which makes it testable, reviewable, and IDE-friendly. It now lives at the Apache Software Foundation (graduated from DAGWorks' stewardship), and the ecosystem includes drivers for common data tools plus integrations for tracking experiments and serving features.

The honest caveat: Hamilton is deliberately lightweight — it is not Airflow or Dagster, so if you need distributed scheduling, retries-as-a-service, or a hosted control plane, you'll layer something on top. It shines for in-process dataflows and feature engineering where full orchestrators are overkill.

1. pip install sf-hamilton (the Apache-era package name).
2. Write your dataflow as Python functions with type-annotated parameters.
3. Build a driver.Driver with your function modules and call driver.execute(["output_name"], inputs={...}).
4. Visualize the DAG with driver.visualize_execution() to inspect lineage.