BaseFramePartition

The class is base for any partition class of pandas backend and serves as the last level on which operations that were conveyed from the partition manager are being performed on an individual block partition.

The class provides an API that has to be overridden by child classes in order to manipulate on data and metadata they store.

The public API exposed by the children of this class is used in BaseFrameManager.

The objects wrapped by the child classes are treated as immutable by BaseFrameManager subclasses and no logic for updating inplace.

Public API

class modin.engines.base.frame.partition.BaseFramePartition

An abstract class that is base for any partition class of pandas backend.

The class providing an API that has to be overridden by child classes.

add_to_apply_calls(func, **kwargs)

Add a function to the call queue.

Parameters:
  • func (callable) – Function to be added to the call queue.
  • **kwargs (dict) – Additional keyword arguments to be passed in func.
Returns:

New BaseFramePartition object with the function added to the call queue.

Return type:

BaseFramePartition

Notes

This function will be executed when apply is called. It will be executed in the order inserted; apply’s func operates the last and return.

apply(func, **kwargs)

Apply a function to the object wrapped by this partition.

Parameters:
  • func (callable) – Function to apply.
  • **kwargs (dict) – Additional keyword arguments to be passed in func.
Returns:

New BaseFramePartition object.

Return type:

BaseFramePartition

Notes

It is up to the implementation how kwargs are handled. They are an important part of many implementations. As of right now, they are not serialized.

drain_call_queue()

Execute all operations stored in the call queue on the object wrapped by this partition.

classmethod empty()

Create a new partition that wraps an empty pandas DataFrame.

Returns:New BaseFramePartition object.
Return type:BaseFramePartition
get()

Get the object wrapped by this partition.

Returns:The object that was wrapped by this partition.
Return type:object

Notes

This is the opposite of the classmethod put. E.g. if you assign x = BaseFramePartition.put(1), x.get() should always return 1.

length()

Get the length of the object wrapped by this partition.

Returns:The length of the object.
Return type:int
classmethod length_extraction_fn()

Return the function that computes the length of the object wrapped by this partition.

Returns:The function that computes the length of the object wrapped by this partition.
Return type:callable
mask(row_indices, col_indices)

Lazily create a mask that extracts the indices provided.

Parameters:
  • row_indices (list-like) – The indices for the rows to extract.
  • col_indices (list-like) – The indices for the columns to extract.
Returns:

New BaseFramePartition object.

Return type:

BaseFramePartition

classmethod preprocess_func(func)

Preprocess a function before an apply call.

Parameters:func (callable) – Function to preprocess.
Returns:An object that can be accepted by apply.
Return type:callable

Notes

This is a classmethod because the definition of how to preprocess should be class-wide. Also, we may want to use this before we deploy a preprocessed function to multiple BaseFramePartition objects.

classmethod put(obj)

Put an object into a store and wrap it with partition object.

Parameters:obj (object) – An object to be put.
Returns:New BaseFramePartition object.
Return type:BaseFramePartition
to_numpy(**kwargs)

Convert the object wrapped by this partition to a NumPy array.

Parameters:**kwargs (dict) – Additional keyword arguments to be passed in to_numpy.
Returns:
Return type:np.ndarray

Notes

If the underlying object is a pandas DataFrame, this will return a 2D NumPy array.

to_pandas()

Convert the object wrapped by this partition to a pandas DataFrame.

Returns:
Return type:pandas.DataFrame

Notes

If the underlying object is a pandas DataFrame, this will likely only need to call get.

wait()

Wait for completion of computations on the object wrapped by the partition.

width()

Get the width of the object wrapped by the partition.

Returns:The width of the object.
Return type:int
classmethod width_extraction_fn()

Return the function that computes the width of the object wrapped by this partition.

Returns:The function that computes the width of the object wrapped by this partition.
Return type:callable