From 4fc32225a92b5de382f71551988cc4e072fb7642 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Sun, 16 Aug 2026 12:15:51 +0800 Subject: [PATCH] audit: sweep Distributed Computing to the shortlist cap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removed luigi (1,546,263 downloads/month, repo active with a push 2026-07) — the drop is editorial: Spotify itself moved to Flyte, 2026 orchestration comparisons no longer name luigi, and the Workflow Orchestration slots in Job Schedulers go to airflow, prefect, dagster. Flattened the sole Batch Processing subcategory — the label added nothing, the section is a single use case. Reordered by tier and downloads: obvious choices ray 62.6M, pyspark 48.3M, dask 30.0M; challengers joblib 216.9M raw (scikit-learn-dependency-inflated, judgment call — raw count still sorts it first in tier) and mpi4py 1.0M (HPC standard, specific audience not small). Co-Authored-By: Claude --- README.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index eb054c54..72081921 100644 --- a/README.md +++ b/README.md @@ -713,13 +713,11 @@ _Software and libraries for DevOps._ _Frameworks and libraries for Distributed Computing._ -- Batch Processing - - [dask](https://github.com/dask/dask) - A flexible parallel computing library for analytic computing. - - [luigi](https://github.com/spotify/luigi) - A module that helps you build complex pipelines of batch jobs. - - [mpi4py](https://github.com/mpi4py/mpi4py) - Python bindings for MPI. - - [pyspark](https://github.com/apache/spark) - [Apache Spark](https://spark.apache.org/) Python API. - - [joblib](https://github.com/joblib/joblib) - A set of tools to provide lightweight pipelining in Python. - - [ray](https://github.com/ray-project/ray/) - A system for parallel and distributed Python that unifies the machine learning ecosystem. +- [ray](https://github.com/ray-project/ray/) - A system for parallel and distributed Python that unifies the machine learning ecosystem. +- [pyspark](https://github.com/apache/spark) - [Apache Spark](https://spark.apache.org/) Python API. +- [dask](https://github.com/dask/dask) - A flexible parallel computing library for analytic computing. +- [joblib](https://github.com/joblib/joblib) - A set of tools to provide lightweight pipelining in Python. +- [mpi4py](https://github.com/mpi4py/mpi4py) - Python bindings for MPI. ### Task Queues