Using Modin in a Cluster#
Often in practice we have a need to exceed the capabilities of a single machine. Modin works and performs well in both local mode and in a cluster environment. The key advantage of Modin is that your notebook does not change between local development and cluster execution. Users are not required to think about how many workers exist or how to distribute and partition their data; Modin handles all of this seamlessly and transparently.
Starting up a Ray Cluster#
pip install boto3
To start up the Ray cluster, run the following command in your terminal:
ray up modin-cluster.yaml
This configuration script starts 1 head node (m5.24xlarge) and 7 workers (m5.24xlarge), 768 total CPUs. For more information on how to launch a Ray cluster across different cloud providers or on-premise, you can also refer to the Ray’s cluster docs.
By default, Modin on Ray uses 60% of the system memory. It is recommended to use the same amount, when using your own cluster (for each node).
Connecting to a Ray Cluster#
To connect to the Ray cluster, run the following command in your terminal:
ray attach modin-cluster.yaml
The following code checks that the Ray cluster is properly configured and attached to Modin:
from modin.config import NPartitions
assert NPartitions.get() == 768, "Not all Ray nodes are started up yet"
Congratualions! You have successfully connected to the Ray cluster.
Using Modin on a Ray Cluster#
Now that we have a Ray cluster up and running, we can use Modin to perform pandas operation as if we were working with pandas on a single machine. We test Modin’s performance on the 200MB NYC Taxi dataset that was provided as part of our Modin’s cluster setup config. We can time the following operation in a Jupyter notebook:
df = pd.read_csv("big_yellow.csv", quoting=3)
count_result = df.count()
groupby_result = df.groupby("passenger_count").count()
apply_result = df.map(str)
When using local paths, make sure that they are available on all nodes in the cluster, for example using distributed file system like NFS.
Modin performance scales as the number of nodes and cores increases. The following chart shows the performance of the above operations with 2, 4, and 8 nodes, with improvements in performance as we increase the number of resources Modin can use.
Advanced: Configuring your Ray Environment#
In some cases, it may be useful to customize your Ray environment. Below, we have listed a few ways you can solve common problems in data management with Modin by customizing your Ray environment. It is possible to use any of Ray’s initialization parameters, which are all found in Ray’s API docs.
import modin.pandas as pd
Modin will automatically connect to the Ray instance that is already running. This way, you can customize your Ray environment for use in Modin!