PandasFramePartitionManager¶

The class is base for any partition manager class of pandas backend and serves as intermediate level between pandas base frame and conforming partition class. The class is responsible for partitions manipulation and applying a function to individual partitions: block partitions, row partitions or column partitions, i.e. the class can form axis partitions from block partitions to apply a function if an operation requires access to an entire column or row. The class translates frame API into partition API and also can have some preprocessing operations depending on the partition type for improving performance (for example, preprocess_func()).

Main task of partition manager is to keep knowledge of how partitions are stored and managed internal to itself, so surrounding code could use it via lean enough API without worrying about implementation details.

Partition manager can apply user-passed (arbitrary) function in different modes:

block-wise (apply a function to individual block partitions):
- optinally accepting partition indices along each axis
- optionally accepting an item to be split so parts of it would be sent to each partition
along a full axis (apply a function to an entire column or row made up of block partitions when user function needs information about the whole axis)

It can also broadcast partitions from right to left when executing certain operations making right partitions available for functions executed where left live.

Partition manager also is used to create “logical” partitions, or axis partitions by joining existing partitions along specified axis (either rows or labels), and to concatenate different partition sets along given axis.

It also maintains mapping from “external” (end user-visible) indices along all axes to internal indices which are actually pairs of indices of partitions and indices inside the partitions, as well as manages conversion to numpy and pandas representations.

Public API¶

class modin.engines.base.frame.partition_manager.PandasFramePartitionManager¶

Base class for managing the dataframe data layout and operators across the distribution of partitions.

Partition class is the class to use for storing each partition. Each partition must extend the PandasFramePartition class.

classmethod apply_func_to_indices_both_axis(partitions, func, row_partitions_list, col_partitions_list, item_to_distribute=None)¶

Apply a function along both axes.

Parameters

partitions (np.ndarray) – The partitions to which the func will apply.
func (callable) – The function to apply.
row_partitions_list (list) – List of row partitions.
col_partitions_list (list) – List of column partitions.
item_to_distribute (item, default: None) – The item to split up so it can be applied over both axes.

Returns

A NumPy array with partitions.

Return type

np.ndarray

Notes

For your func to operate directly on the indices provided, it must use row_internal_indices, col_internal_indices as keyword arguments.

classmethod apply_func_to_select_indices(axis, partitions, func, indices, keep_remaining=False)¶

Apply a function to select indices.

Parameters

axis ({0, 1}) – Axis to apply the func over.
partitions (np.ndarray) – The partitions to which the func will apply.
func (callable) – The function to apply to these indices of partitions.
indices (dict) – The indices to apply the function to.
keep_remaining (bool, default: False) – Whether or not to keep the other partitions. Some operations may want to drop the remaining partitions and keep only the results.

Returns

A NumPy array with partitions.

Return type

np.ndarray

Notes

Your internal function must take a kwarg internal_indices for this to work correctly. This prevents information leakage of the internal index to the external representation.

classmethod apply_func_to_select_indices_along_full_axis(axis, partitions, func, indices, keep_remaining=False)¶

Apply a function to a select subset of full columns/rows.

Parameters

axis ({0, 1}) – The axis to apply the function over.
partitions (np.ndarray) – The partitions to which the func will apply.
func (callable) – The function to apply.
indices (list-like) – The global indices to apply the func to.
keep_remaining (bool, default: False) – Whether or not to keep the other partitions. Some operations may want to drop the remaining partitions and keep only the results.

Returns

A NumPy array with partitions.

Return type

np.ndarray

Notes

This should be used when you need to apply a function that relies on some global information for the entire column/row, but only need to apply a function to a subset. For your func to operate directly on the indices provided, it must use internal_indices as a keyword argument.

classmethod axis_partition(partitions, axis)¶

Logically partition along given axis (columns or rows).

Parameters

partitions (list-like) – List of partitions to be combined.
axis ({0, 1}) – 0 for column partitions, 1 for row partitions.

Returns

A list of BaseFrameAxisPartition objects.

Return type

list

classmethod binary_operation(axis, left, func, right)¶

Apply a function that requires two PandasFrame objects.

Parameters

axis ({0, 1}) – The axis to apply the function over (0 - rows, 1 - columns).
left (np.ndarray) – The partitions of left PandasFrame.
func (callable) – The function to apply.
right (np.ndarray) – The partitions of right PandasFrame.

Returns

A NumPy array with new partitions.

Return type

np.ndarray

classmethod broadcast_apply(axis, apply_func, left, right, other_name='r')¶

Broadcast the right partitions to left and apply apply_func function.

Parameters

axis ({0, 1}) – Axis to apply and broadcast over.
apply_func (callable) – Function to apply.
left (NumPy 2D array) – Left partitions.
right (NumPy 2D array) – Right partitions.
other_name (str, default: "r") – Name of key-value argument for apply_func that is used to pass right to apply_func.

Returns

An of partition objects.

Return type

NumPy array

Notes

This will often be overridden by implementations. It materializes the entire partitions of the right and applies them to the left through apply.

classmethod broadcast_apply_select_indices(axis, apply_func, left, right, left_indices, right_indices, keep_remaining=False)¶

Broadcast the right partitions to left and apply apply_func to selected indices.

Parameters

axis ({0, 1}) – Axis to apply and broadcast over.
apply_func (callable) – Function to apply.
left (NumPy 2D array) – Left partitions.
right (NumPy 2D array) – Right partitions.
left_indices (list-like) – Indices to apply function to.
right_indices (dictionary of indices of right partitions) – Indices that you want to bring at specified left partition, for example dict {key: {key1: [0, 1], key2: [5]}} means that in left[key] you want to broadcast [right[key1], right[key2]] partitions and internal indices for right must be [[0, 1], [5]].
keep_remaining (bool, default: False) – Whether or not to keep the other partitions. Some operations may want to drop the remaining partitions and keep only the results.

Returns

An array of partition objects.

Return type

NumPy array

Notes

Your internal function must take these kwargs: [internal_indices, other, internal_other_indices] to work correctly!

classmethod broadcast_axis_partitions(axis, apply_func, left, right, keep_partitioning=False, apply_indices=None, enumerate_partitions=False, lengths=None)¶

Broadcast the right partitions to left and apply apply_func along full axis.

Parameters

axis ({0, 1}) – Axis to apply and broadcast over.
apply_func (callable) – Function to apply.
left (NumPy 2D array) – Left partitions.
right (NumPy 2D array) – Right partitions.
keep_partitioning (boolean, default: False) – The flag to keep partition boundaries for Modin Frame. Setting it to True disables shuffling data from one partition to another.
apply_indices (list of ints, default: None) – Indices of axis ^ 1 to apply function over.
enumerate_partitions (bool, default: False) – Whether or not to pass partition index into apply_func. Note that apply_func must be able to accept partition_idx kwarg.
lengths (list of ints, default: None) – The list of lengths to shuffle the object.

Returns

An array of partition objects.

Return type

NumPy array

classmethod column_partitions(partitions)¶

Get the list of BaseFrameAxisPartition objects representing column-wise paritions.

Parameters: partitions (list-like) – List of (smaller) partitions to be combined to column-wise partitions.
Returns: A list of BaseFrameAxisPartition objects.
Return type: list

Notes

Each value in this list will be an BaseFrameAxisPartition object. BaseFrameAxisPartition is located in axis_partition.py.

classmethod concat(axis, left_parts, right_parts)¶

Concatenate the blocks of partitions with another set of blocks.

Parameters

axis (int) – The axis to concatenate to.
left_parts (np.ndarray) – NumPy array of partitions to concatenate with.
right_parts (np.ndarray or list) – NumPy array of partitions to be concatenated.

Returns

A new NumPy array with concatenated partitions.

Return type

np.ndarray

Notes

Assumes that the blocks are already the same shape on the dimension being concatenated. A ValueError will be thrown if this condition is not met.

classmethod concatenate(dfs)¶

Concatenate pandas DataFrames with saving ‘category’ dtype.

Parameters: dfs (list) – List of pandas DataFrames to concatenate.
Returns: A pandas DataFrame
Return type: pandas.DataFrame

classmethod finalize(partitions)¶

Perform all deferred calls on partitions.

Parameters: partitions (np.ndarray) – Partitions of Modin Dataframe on which all deferred calls should be performed.

classmethod from_arrow(at, return_dims=False)¶

Return the partitions from Apache Arrow (PyArrow).

Parameters

at (pyarrow.table) – Arrow Table.
return_dims (bool, default: False) – If it’s True, return as (np.ndarray, row_lengths, col_widths), else np.ndarray.

Returns

A NumPy array with partitions (with dimensions or not).

Return type

np.ndarray or (np.ndarray, row_lengths, col_widths)

classmethod from_pandas(df, return_dims=False)¶

Return the partitions from pandas.DataFrame.

Parameters

df (pandas.DataFrame) – A pandas.DataFrame.
return_dims (bool, default: False) – If it’s True, return as (np.ndarray, row_lengths, col_widths), else np.ndarray.

Returns

A NumPy array with partitions (with dimensions or not).

Return type

np.ndarray or (np.ndarray, row_lengths, col_widths)

classmethod get_indices(axis, partitions, index_func=None)¶

Get the internal indices stored in the partitions.

Parameters

axis ({0, 1}) – Axis to extract the labels over.
partitions (np.ndarray) – NumPy array with PandasFramePartition’s.
index_func (callable, default: None) – The function to be used to extract the indices.

Returns

A pandas Index object.

Return type

pandas.Index

Notes

These are the global indices of the object. This is mostly useful when you have deleted rows/columns internally, but do not know which ones were deleted.

classmethod groupby_reduce(axis, partitions, by, map_func, reduce_func, apply_indices=None)¶

Groupby data using the map_func provided along the axis over the partitions then reduce using reduce_func.

Parameters

axis ({0, 1}) – Axis to groupby over.
partitions (NumPy 2D array) – Partitions of the ModinFrame to groupby.
by (NumPy 2D array) – Partitions of ‘by’ to broadcast.
map_func (callable) – Map function.
reduce_func (callable,) – Reduce function.
apply_indices (list of ints, default: None) – Indices of axis ^ 1 to apply function over.

Returns

Partitions with applied groupby.

Return type

NumPy array

classmethod lazy_map_partitions(partitions, map_func)¶

Apply map_func to every partition in partitions lazily.

Parameters

partitions (NumPy 2D array) – Partitions of Modin Frame.
map_func (callable) – Function to apply.

Returns

An array of partitions

Return type

NumPy array

classmethod map_axis_partitions(axis, partitions, map_func, keep_partitioning=False, lengths=None, enumerate_partitions=False)¶

Apply map_func to every partition in partitions along given axis.

Parameters

axis ({0, 1}) – Axis to perform the map across (0 - index, 1 - columns).
partitions (NumPy 2D array) – Partitions of Modin Frame.
map_func (callable) – Function to apply.
keep_partitioning (bool, default: False) – Whether to keep partitioning for Modin Frame. Setting it to True stops data shuffling between partitions.
lengths (list of ints, default: None) – List of lengths to shuffle the object.
enumerate_partitions (bool, default: False) – Whether or not to pass partition index into map_func. Note that map_func must be able to accept partition_idx kwarg.

Returns

An array of new partitions for Modin Frame.

Return type

NumPy array

Notes

This method should be used in the case when map_func relies on some global information about the axis.

classmethod map_partitions(partitions, map_func)¶

Apply map_func to every partition in partitions.

Parameters

partitions (NumPy 2D array) – Partitions housing the data of Modin Frame.
map_func (callable) – Function to apply.

Returns

An array of partitions

Return type

NumPy array

classmethod preprocess_func(map_func)¶

Preprocess a function to be applied to PandasFramePartition objects.

Parameters: map_func (callable) – The function to be preprocessed.
Returns: The preprocessed version of the map_func provided.
Return type: callable

Notes

Preprocessing does not require any specific format, only that the PandasFramePartition.apply method will recognize it (for the subclass being used).

If your PandasFramePartition objects assume that a function provided is serialized or wrapped or in some other format, this is the place to add that logic. It is possible that this can also just return map_func if the apply method of the PandasFramePartition object you are using does not require any modification to a given function.

classmethod row_partitions(partitions)¶

List of BaseFrameAxisPartition objects representing row-wise partitions.

Parameters: partitions (list-like) – List of (smaller) partitions to be combined to row-wise partitions.
Returns: A list of BaseFrameAxisPartition objects.
Return type: list

Notes

Each value in this list will an BaseFrameAxisPartition object. BaseFrameAxisPartition is located in axis_partition.py.

classmethod simple_shuffle(axis, partitions, map_func, lengths)¶

Shuffle data so lengths of partitions match given lengths via calling map_func.

Parameters

axis ({0, 1}) – Axis to perform the map across (0 - index, 1 - columns).
partitions (NumPy 2D array) – Partitions of Modin Frame.
map_func (callable) – Function to apply.
lengths (list(int)) – List of lengths to shuffle the object.

Returns

An array of new partitions for a Modin Frame.

Return type

NumPy array

classmethod to_numpy(partitions, **kwargs)¶

Convert NumPy array of PandasFramePartition to NumPy array of data stored within partitions.

Parameters

partitions (np.ndarray) – NumPy array of PandasFramePartition.
**kwargs (dict) – Keyword arguments for PandasFramePartition.to_numpy function.

Returns

A NumPy array.

Return type

np.ndarray

classmethod to_pandas(partitions)¶

Convert NumPy array of PandasFramePartition to pandas DataFrame.

Parameters: partitions (np.ndarray) – NumPy array of PandasFramePartition.
Returns: A pandas DataFrame
Return type: pandas.DataFrame