Pachyderm
Git-like data versioning and reproducible ML pipelines on Kubernetes.
Acquired — still operating
acquired by HPE (announced Jan 2023)
About
Pachyderm combined data versioning with pipeline execution: datasets are versioned like Git commits, and Kubernetes-native pipelines automatically reprocess only what changed when data updates, giving full lineage from raw data to trained model. That provenance model made it a natural fit for ML teams that need reproducibility for compliance and debugging, with integrations across the labeling and training ecosystem (Label Studio, Toloka, MLflow, Spark). The San Francisco company raised $28.1M from investors before being acquired.
In January 2023 Hewlett Packard Enterprise announced the acquisition of Pachyderm to anchor its AI-at-scale portfolio, folding reproducible pipeline automation into HPE's HPC and AI offerings. The entry is historical as an independent product; its concepts (data versioning plus execution, incremental processing) live on in HPE's platform and influenced tools like DVC and lakeFS.
What it can do
- Git-like versioning of datasets with content-addressed storage
- Kubernetes-native pipelines with automatic incremental reprocessing
- End-to-end data lineage from raw data to deployed model
- Integrations with labeling tools, Spark, MLflow and data warehouses
Setup
Historically: Helm chart onto a Kubernetes cluster; now via HPE's AI offerings.
Prerequisites
Kubernetes cluster and object storage (for the original OSS deployment).
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 89; Trending score 70/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.