Medium data and out-of-core data tools

Python fast out-of-core data frames

Cross-language data frames built in C++

larger-than-RAM data manipulation library in R

Python pandas-API compatible out-of-core computation library.

There is feedback that it's not very stable and doesn't play well in the Kubernetes ecosystem. Hence, Spark was chosen over this.

JuliaDB is the only Julia data tool for larger-than-RAM data.

data storage format. Heavily R-based but has cross language support coming. Built in C++

A straight dump of Apache Arrow in-memory format onto disk. Has support in R, Python, and Julia.

In-memory data specification

It explains some interesting points about what needs to be done to make an outstanding distributed data processing platform