PyArrow Parsers Module Description

This module houses parser classes that are responsible for data parsing on the workers for the PyArrow backend. Parsers for PyArrow backends follow an interface of pandas backend parsers: parser class of every file format implements parse method, which parses the specified part of the file and builds PyArrow tables from the parsed data, based on the specified chunk size and number of splits. The resulted PyArrow tables will be used as a partitions payload in the PyarrowOnRayFrame.

Public API

Module houses Modin parser classes, that are used for data parsing on the workers.

class modin.backends.pyarrow.parsers.PyarrowCSVParser

Class for handling CSV files on the workers using PyArrow backend.

parse(fname, num_splits, start, end, header, **kwargs)

Parse CSV file into PyArrow tables.

Parameters

  • fname (str) – Name of the CSV file to parse.

  • num_splits (int) – Number of partitions to split the resulted PyArrow table into.

  • start (int) – Position in the specified file to start parsing from.

  • end (int) – Position in the specified file to end parsing at.

  • header (str) – Header line that will be interpret as the first line of the parsed CSV file.

  • **kwargs (kwargs) – Serves the compatibility purpose. Does not affect the result.

Returns

List with splitted parse results and it’s metadata:

  • First num_split elements are PyArrow tables, representing the corresponding chunk.

  • Next element is the number of rows in the parsed table.

  • Last element is the pandas Series, containing the data-types for each column of the parsed table.

Return type

list