Base pandas Dataset API#
The class implements functionality that is common to Modin’s pandas API for both DataFrame
and Series
classes.
Public API#
- class modin.pandas.base.BasePandasDataset
Implement most of the common code that exists in DataFrame/Series.
Since both objects share the same underlying representation, and the algorithms are the same, we use this object to define the general behavior of those objects and then use those objects to define the output type.
Notes
See pandas API documentation for pandas.DataFrame, pandas.Series for more.
- abs() Self
Return a BasePandasDataset with absolute numeric value of each element.
Notes
See pandas API documentation for pandas.DataFrame.abs, pandas.Series.abs for more.
- add(other, axis='columns', level=None, fill_value=None) Self
Return addition of BasePandasDataset and other, element-wise (binary operator add).
Notes
See pandas API documentation for pandas.DataFrame.add, pandas.Series.add for more.
- agg(func=None, axis=0, *args, **kwargs) DataFrame | Series | Scalar
Aggregate using one or more operations over the specified axis.
Notes
See pandas API documentation for pandas.DataFrame.agg, pandas.Series.agg for more.
- aggregate(func=None, axis=0, *args, **kwargs) DataFrame | Series | Scalar
Aggregate using one or more operations over the specified axis.
Notes
See pandas API documentation for pandas.DataFrame.aggregate, pandas.Series.aggregate for more.
- align(other, join='outer', axis=None, level=None, copy=None, fill_value=None, method=_NoDefault.no_default, limit=_NoDefault.no_default, fill_axis=_NoDefault.no_default, broadcast_axis=_NoDefault.no_default) tuple[Self, Self]
Align two objects on their axes with the specified join method.
Notes
See pandas API documentation for pandas.DataFrame.align, pandas.Series.align for more.
- all(axis=0, bool_only=False, skipna=True, **kwargs) Self
Return whether all elements are True, potentially over an axis.
Notes
See pandas API documentation for pandas.DataFrame.all, pandas.Series.all for more.
- any(*, axis=0, bool_only=False, skipna=True, **kwargs) Self
Return whether any element is True, potentially over an axis.
Notes
See pandas API documentation for pandas.DataFrame.any, pandas.Series.any for more.
- apply(func, axis, raw, result_type, args, **kwds) BaseQueryCompiler
Apply a function along an axis of the BasePandasDataset.
Notes
See pandas API documentation for pandas.DataFrame.apply, pandas.Series.apply for more.
- asfreq(freq, method=None, how=None, normalize=False, fill_value=None) Self
Convert time series to specified frequency.
Notes
See pandas API documentation for pandas.DataFrame.asfreq, pandas.Series.asfreq for more.
- asof(where, subset=None) Self
Return the last row(s) without any NaNs before where.
Notes
See pandas API documentation for pandas.DataFrame.asof, pandas.Series.asof for more.
- astype(dtype, copy=None, errors='raise') Self
Cast a Modin object to a specified dtype dtype.
Notes
See pandas API documentation for pandas.DataFrame.astype, pandas.Series.astype for more.
- property at: _LocIndexer
Get a single value for a row/column label pair.
Notes
See pandas API documentation for pandas.DataFrame.at, pandas.Series.at for more.
- at_time(time, asof=False, axis=None) Self
Select values at particular time of day (e.g., 9:30AM).
Notes
See pandas API documentation for pandas.DataFrame.at_time, pandas.Series.at_time for more.
- backfill(*, axis=None, inplace=False, limit=None, downcast=_NoDefault.no_default) Self
Synonym for DataFrame.bfill.
Notes
See pandas API documentation for pandas.DataFrame.backfill, pandas.Series.backfill for more.
- between_time(start_time, end_time, inclusive='both', axis=None) Self
Select values between particular times of the day (e.g., 9:00-9:30 AM).
By setting
start_time
to be later thanend_time
, you can get the times that are not between the two times.- Parameters:
start_time (datetime.time or str) – Initial time as a time filter limit.
end_time (datetime.time or str) – End time as a time filter limit.
inclusive ({"both", "neither", "left", "right"}, default "both") – Include boundaries; whether to set each bound as closed or open.
axis ({0 or 'index', 1 or 'columns'}, default 0) – Determine range time on index or columns value. For Series this parameter is unused and defaults to 0.
- Returns:
Data from the original object filtered to the specified dates range.
- Return type:
- Raises:
TypeError – If the index is not a
DatetimeIndex
See also
at_time
Select values at a particular time of the day.
first
Select initial periods of time series based on a date offset.
last
Select final periods of time series based on a date offset.
DatetimeIndex.indexer_between_time
Get just the index locations for values between particular times of the day.
Examples
>>> i = pd.date_range('2018-04-09', periods=4, freq='1D20min') >>> ts = pd.DataFrame({'A': [1, 2, 3, 4]}, index=i) >>> ts A 2018-04-09 00:00:00 1 2018-04-10 00:20:00 2 2018-04-11 00:40:00 3 2018-04-12 01:00:00 4
>>> ts.between_time('0:15', '0:45') A 2018-04-10 00:20:00 2 2018-04-11 00:40:00 3
You get the times that are not between two times by setting
start_time
later thanend_time
:>>> ts.between_time('0:45', '0:15') A 2018-04-09 00:00:00 1 2018-04-12 01:00:00 4
Notes
See pandas API documentation for pandas.DataFrame.between_time for more.
- bfill(*, axis=None, inplace=False, limit=None, limit_area=None, downcast=_NoDefault.no_default) Self
Synonym for DataFrame.fillna with
method='bfill'
.Notes
See pandas API documentation for pandas.DataFrame.bfill, pandas.Series.bfill for more.
- bool() bool
Return the bool of a single element BasePandasDataset.
Notes
See pandas API documentation for pandas.DataFrame.bool, pandas.Series.bool for more.
- clip(lower=None, upper=None, *, axis=None, inplace=False, **kwargs) Self
Trim values at input threshold(s).
Notes
See pandas API documentation for pandas.DataFrame.clip, pandas.Series.clip for more.
- combine(other, func, fill_value=None, **kwargs) Self
Perform combination of BasePandasDataset-s according to func.
Notes
See pandas API documentation for pandas.DataFrame.combine, pandas.Series.combine for more.
- combine_first(other) Self
Update null elements with value in the same location in other.
Notes
See pandas API documentation for pandas.DataFrame.combine_first, pandas.Series.combine_first for more.
- convert_dtypes(infer_objects: bool = True, convert_string: bool = True, convert_integer: bool = True, convert_boolean: bool = True, convert_floating: bool = True, dtype_backend: DtypeBackend = 'numpy_nullable') Self
Convert columns to best possible dtypes using dtypes supporting
pd.NA
.Notes
See pandas API documentation for pandas.DataFrame.convert_dtypes, pandas.Series.convert_dtypes for more.
- copy(deep=True) Self
Make a copy of the object’s metadata.
Notes
See pandas API documentation for pandas.DataFrame.copy, pandas.Series.copy for more.
- count(axis=0, numeric_only=False) Series | Scalar
Count non-NA cells for BasePandasDataset.
Notes
See pandas API documentation for pandas.DataFrame.count, pandas.Series.count for more.
- cummax(axis=None, skipna=True, *args, **kwargs) Self
Return cumulative maximum over a BasePandasDataset axis.
Notes
See pandas API documentation for pandas.DataFrame.cummax, pandas.Series.cummax for more.
- cummin(axis=None, skipna=True, *args, **kwargs) Self
Return cumulative minimum over a BasePandasDataset axis.
Notes
See pandas API documentation for pandas.DataFrame.cummin, pandas.Series.cummin for more.
- cumprod(axis=None, skipna=True, *args, **kwargs) Self
Return cumulative product over a BasePandasDataset axis.
Notes
See pandas API documentation for pandas.DataFrame.cumprod, pandas.Series.cumprod for more.
- cumsum(axis=None, skipna=True, *args, **kwargs) Self
Return cumulative sum over a BasePandasDataset axis.
Notes
See pandas API documentation for pandas.DataFrame.cumsum, pandas.Series.cumsum for more.
- describe(percentiles=None, include=None, exclude=None) Self
Generate descriptive statistics.
Notes
See pandas API documentation for pandas.DataFrame.describe, pandas.Series.describe for more.
- diff(periods=1, axis=0) Self
First discrete difference of element.
Notes
See pandas API documentation for pandas.DataFrame.diff, pandas.Series.diff for more.
- div(other, axis='columns', level=None, fill_value=None) Self
Get floating division of BasePandasDataset and other, element-wise (binary operator truediv).
Notes
See pandas API documentation for pandas.DataFrame.div, pandas.Series.div for more.
- divide(other, axis='columns', level=None, fill_value=None) Self
Get floating division of BasePandasDataset and other, element-wise (binary operator truediv).
Notes
See pandas API documentation for pandas.DataFrame.divide, pandas.Series.divide for more.
- drop(labels=None, *, axis=0, index=None, columns=None, level=None, inplace=False, errors='raise') Self
Drop specified labels from BasePandasDataset.
Notes
See pandas API documentation for pandas.DataFrame.drop, pandas.Series.drop for more.
- drop_duplicates(keep='first', inplace=False, **kwargs) Self
Return BasePandasDataset with duplicate rows removed.
Notes
See pandas API documentation for pandas.DataFrame.drop_duplicates, pandas.Series.drop_duplicates for more.
- droplevel(level, axis=0) Self
Return BasePandasDataset with requested index / column level(s) removed.
Notes
See pandas API documentation for pandas.DataFrame.droplevel, pandas.Series.droplevel for more.
- dropna(*, axis: Axis = 0, how: str | lib.NoDefault = _NoDefault.no_default, thresh: int | lib.NoDefault = _NoDefault.no_default, subset: IndexLabel = None, inplace: bool = False, ignore_index: bool = False) Self
Remove missing values.
Notes
See pandas API documentation for pandas.DataFrame.dropna, pandas.Series.dropna for more.
- eq(other, axis='columns', level=None) Self
Get equality of BasePandasDataset and other, element-wise (binary operator eq).
Notes
See pandas API documentation for pandas.DataFrame.eq, pandas.Series.eq for more.
- ewm(com: float | None = None, span: float | None = None, halflife: float | TimedeltaConvertibleTypes | None = None, alpha: float | None = None, min_periods: int | None = 0, adjust: bool = True, ignore_na: bool = False, axis: Axis = _NoDefault.no_default, times: str | np.ndarray | BasePandasDataset | None = None, method: str = 'single') pandas.core.window.ewm.ExponentialMovingWindow
Provide exponentially weighted (EW) calculations.
Notes
See pandas API documentation for pandas.DataFrame.ewm, pandas.Series.ewm for more.
- expanding(min_periods=1, axis=_NoDefault.no_default, method='single') Expanding
Provide expanding window calculations.
Notes
See pandas API documentation for pandas.DataFrame.expanding, pandas.Series.expanding for more.
- explode(column, ignore_index: bool = False) Self
Transform each element of a list-like to a row.
Notes
See pandas API documentation for pandas.DataFrame.explode, pandas.Series.explode for more.
- ffill(*, axis=None, inplace=False, limit=None, limit_area=None, downcast=_NoDefault.no_default) Self | None
Synonym for DataFrame.fillna with
method='ffill'
.Notes
See pandas API documentation for pandas.DataFrame.ffill, pandas.Series.ffill for more.
- fillna(squeeze_self, squeeze_value, value=None, method=None, axis=None, inplace=False, limit=None, downcast=_NoDefault.no_default) Self | None
Fill NA/NaN values using the specified method.
- Parameters:
squeeze_self (bool) – If True then self contains a Series object, if False then self contains a DataFrame object.
squeeze_value (bool) – If True then value contains a Series object, if False then value contains a DataFrame object.
value (scalar, dict, Series, or DataFrame, default: None) – Value to use to fill holes (e.g. 0), alternately a dict/Series/DataFrame of values specifying which value to use for each index (for a Series) or column (for a DataFrame). Values not in the dict/Series/DataFrame will not be filled. This value cannot be a list.
method ({'backfill', 'bfill', 'pad', 'ffill', None}, default: None) – Method to use for filling holes in reindexed Series pad / ffill: propagate last valid observation forward to next valid backfill / bfill: use next valid observation to fill gap.
axis ({None, 0, 1}, default: None) – Axis along which to fill missing values.
inplace (bool, default: False) – If True, fill in-place. Note: this will modify any other views on this object (e.g., a no-copy slice for a column in a DataFrame).
limit (int, default: None) – If method is specified, this is the maximum number of consecutive NaN values to forward/backward fill. In other words, if there is a gap with more than this number of consecutive NaNs, it will only be partially filled. If method is not specified, this is the maximum number of entries along the entire axis where NaNs will be filled. Must be greater than 0 if not None.
downcast (dict, default: None) – A dict of item->dtype of what to downcast if possible, or the string ‘infer’ which will try to downcast to an appropriate equal type (e.g. float64 to int64 if possible).
- Returns:
Object with missing values filled or None if
inplace=True
.- Return type:
Notes
See pandas API documentation for pandas.DataFrame.fillna, pandas.Series.fillna for more.
- filter(items=None, like=None, regex=None, axis=None) Self
Subset the BasePandasDataset rows or columns according to the specified index labels.
Notes
See pandas API documentation for pandas.DataFrame.filter, pandas.Series.filter for more.
- first(offset) Self | None
Select initial periods of time series data based on a date offset.
Notes
See pandas API documentation for pandas.DataFrame.first, pandas.Series.first for more.
- first_valid_index() int
Return index for first non-NA value or None, if no non-NA value is found.
Notes
See pandas API documentation for pandas.DataFrame.first_valid_index, pandas.Series.first_valid_index for more.
- property flags
Get the properties associated with this pandas object.
The available flags are
Flags.allows_duplicate_labels
See also
Flags
Flags that apply to pandas objects.
DataFrame.attrs
Global metadata applying to this dataset.
Notes
See pandas API documentation for pandas.DataFrame.flags, pandas.Series.flags for more. “Flags” differ from “metadata”. Flags reflect properties of the pandas object (the Series or DataFrame). Metadata refer to properties of the dataset, and should be stored in
DataFrame.attrs
.Examples
>>> df = pd.DataFrame({"A": [1, 2]}) >>> df.flags <Flags(allows_duplicate_labels=True)>
Flags can be get or set using
.
>>> df.flags.allows_duplicate_labels True >>> df.flags.allows_duplicate_labels = False
Or by slicing with a key
>>> df.flags["allows_duplicate_labels"] False >>> df.flags["allows_duplicate_labels"] = True
- floordiv(other, axis='columns', level=None, fill_value=None) Self
Get integer division of BasePandasDataset and other, element-wise (binary operator floordiv).
Notes
See pandas API documentation for pandas.DataFrame.floordiv, pandas.Series.floordiv for more.
- ge(other, axis='columns', level=None) Self
Get greater than or equal comparison of BasePandasDataset and other, element-wise (binary operator ge).
Notes
See pandas API documentation for pandas.DataFrame.ge, pandas.Series.ge for more.
- get(key, default=None) DataFrame | Series | Scalar
Get item from object for given key.
Notes
See pandas API documentation for pandas.DataFrame.get, pandas.Series.get for more.
- get_backend() str
Get the backend for this
BasePandasDataset
.- Returns:
The name of the backend.
- Return type:
str
- gt(other, axis='columns', level=None) Self
Get greater than comparison of BasePandasDataset and other, element-wise (binary operator gt).
Notes
See pandas API documentation for pandas.DataFrame.gt, pandas.Series.gt for more.
- head(n=5) Self
Return the first n rows.
Notes
See pandas API documentation for pandas.DataFrame.head, pandas.Series.head for more.
- property iat: _iLocIndexer
Get a single value for a row/column pair by integer position.
Notes
See pandas API documentation for pandas.DataFrame.iat, pandas.Series.iat for more.
- idxmax(axis=0, skipna=True, numeric_only=False) Self
Return index of first occurrence of maximum over requested axis.
Notes
See pandas API documentation for pandas.DataFrame.idxmax, pandas.Series.idxmax for more.
- idxmin(axis=0, skipna=True, numeric_only=False) Self
Return index of first occurrence of minimum over requested axis.
Notes
See pandas API documentation for pandas.DataFrame.idxmin, pandas.Series.idxmin for more.
- property iloc: _iLocIndexer
Purely integer-location based indexing for selection by position.
Notes
See pandas API documentation for pandas.DataFrame.iloc, pandas.Series.iloc for more.
- property index: Index
Get the index for this DataFrame.
- Returns:
The union of all indexes across the partitions.
- Return type:
pandas.Index
- infer_objects(copy=None) Self
Attempt to infer better dtypes for object columns.
Notes
See pandas API documentation for pandas.DataFrame.infer_objects, pandas.Series.infer_objects for more.
- interpolate(method='linear', *, axis=0, limit=None, inplace=False, limit_direction: Optional[str] = None, limit_area=None, downcast=_NoDefault.no_default, **kwargs) Self
Fill NaN values using an interpolation method.
Please note that only
method='linear'
is supported for DataFrame/Series with a MultiIndex.- Parameters:
method (str, default 'linear') –
Interpolation technique to use. One of:
’linear’: Ignore the index and treat the values as equally spaced. This is the only method supported on MultiIndexes.
’time’: Works on daily and higher resolution data to interpolate given length of interval.
’index’, ‘values’: use the actual numerical values of the index.
’pad’: Fill in NaNs using existing values.
’nearest’, ‘zero’, ‘slinear’, ‘quadratic’, ‘cubic’, ‘barycentric’, ‘polynomial’: Passed to scipy.interpolate.interp1d, whereas ‘spline’ is passed to scipy.interpolate.UnivariateSpline. These methods use the numerical values of the index. Both ‘polynomial’ and ‘spline’ require that you also specify an order (int), e.g.
df.interpolate(method='polynomial', order=5)
. Note that, slinear method in Pandas refers to the Scipy first order spline instead of Pandas first order spline.’krogh’, ‘piecewise_polynomial’, ‘spline’, ‘pchip’, ‘akima’, ‘cubicspline’: Wrappers around the SciPy interpolation methods of similar names. See Notes.
’from_derivatives’: Refers to scipy.interpolate.BPoly.from_derivatives.
axis ({{0 or 'index', 1 or 'columns', None}}, default None) – Axis to interpolate along. For Series this parameter is unused and defaults to 0.
limit (int, optional) – Maximum number of consecutive NaNs to fill. Must be greater than 0.
inplace (bool, default False) – Update the data in place if possible.
limit_direction ({{'forward', 'backward', 'both'}}, Optional) –
Consecutive NaNs will be filled in this direction.
- If limit is specified:
If ‘method’ is ‘pad’ or ‘ffill’, ‘limit_direction’ must be ‘forward’.
If ‘method’ is ‘backfill’ or ‘bfill’, ‘limit_direction’ must be ‘backwards’.
- If ‘limit’ is not specified:
If ‘method’ is ‘backfill’ or ‘bfill’, the default is ‘backward’
else the default is ‘forward’
- raises ValueError if limit_direction is ‘forward’ or ‘both’ and
method is ‘backfill’ or ‘bfill’.
- raises ValueError if limit_direction is ‘backward’ or ‘both’ and
method is ‘pad’ or ‘ffill’.
limit_area ({{None, ‘inside’, ‘outside’}}, default None) –
If limit is specified, consecutive NaNs will be filled with this restriction.
None
: No fill restriction.’inside’: Only fill NaNs surrounded by valid values (interpolate).
’outside’: Only fill NaNs outside valid values (extrapolate).
downcast (optional, 'infer' or None, defaults to None) –
Downcast dtypes if possible.
Deprecated since version 2.1.0.
**kwargs (optional) – Keyword arguments to pass on to the interpolating function.
- Returns:
Returns the same object type as the caller, interpolated at some or all
NaN
values or None ifinplace=True
.- Return type:
See also
fillna
Fill missing values using different methods.
scipy.interpolate.Akima1DInterpolator
Piecewise cubic polynomials (Akima interpolator).
scipy.interpolate.BPoly.from_derivatives
Piecewise polynomial in the Bernstein basis.
scipy.interpolate.interp1d
Interpolate a 1-D function.
scipy.interpolate.KroghInterpolator
Interpolate polynomial (Krogh interpolator).
scipy.interpolate.PchipInterpolator
PCHIP 1-d monotonic cubic interpolation.
scipy.interpolate.CubicSpline
Cubic spline data interpolator.
Notes
See pandas API documentation for pandas.DataFrame.interpolate, pandas.Series.interpolate for more. The ‘krogh’, ‘piecewise_polynomial’, ‘spline’, ‘pchip’ and ‘akima’ methods are wrappers around the respective SciPy implementations of similar names. These use the actual numerical values of the index. For more information on their behavior, see the SciPy documentation.
Examples
Filling in
NaN
in aSeries
via linear interpolation.>>> s = pd.Series([0, 1, np.nan, 3]) >>> s 0 0.0 1 1.0 2 NaN 3 3.0 dtype: float64 >>> s.interpolate() 0 0.0 1 1.0 2 2.0 3 3.0 dtype: float64
Filling in
NaN
in a Series via polynomial interpolation or splines: Both ‘polynomial’ and ‘spline’ methods require that you also specify anorder
(int).>>> s = pd.Series([0, 2, np.nan, 8]) >>> s.interpolate(method='polynomial', order=2) 0 0.000000 1 2.000000 2 4.666667 3 8.000000 dtype: float64
Fill the DataFrame forward (that is, going down) along each column using linear interpolation.
Note how the last entry in column ‘a’ is interpolated differently, because there is no entry after it to use for interpolation. Note how the first entry in column ‘b’ remains
NaN
, because there is no entry before it to use for interpolation.>>> df = pd.DataFrame([(0.0, np.nan, -1.0, 1.0), ... (np.nan, 2.0, np.nan, np.nan), ... (2.0, 3.0, np.nan, 9.0), ... (np.nan, 4.0, -4.0, 16.0)], ... columns=list('abcd')) >>> df a b c d 0 0.0 NaN -1.0 1.0 1 NaN 2.0 NaN NaN 2 2.0 3.0 NaN 9.0 3 NaN 4.0 -4.0 16.0 >>> df.interpolate(method='linear', limit_direction='forward', axis=0) a b c d 0 0.0 NaN -1.0 1.0 1 1.0 2.0 -2.0 5.0 2 2.0 3.0 -3.0 9.0 3 2.0 4.0 -4.0 16.0
Using polynomial interpolation.
>>> df['d'].interpolate(method='polynomial', order=2) 0 1.0 1 4.0 2 9.0 3 16.0 Name: d, dtype: float64
- isin(values) Self
Whether elements in BasePandasDataset are contained in values.
Notes
See pandas API documentation for pandas.DataFrame.isin, pandas.Series.isin for more.
- isna() Self
Detect missing values.
Notes
See pandas API documentation for pandas.DataFrame.isna, pandas.Series.isna for more.
- isnull() Self
Detect missing values.
Notes
See pandas API documentation for pandas.DataFrame.isnull, pandas.Series.isnull for more.
- kurt(axis=0, skipna=True, numeric_only=False, **kwargs) Series | float
Return unbiased kurtosis over requested axis.
Kurtosis obtained using Fisher’s definition of kurtosis (kurtosis of normal == 0.0). Normalized by N-1.
- Parameters:
axis ({index (0), columns (1)}) –
Axis for the function to be applied on. For Series this parameter is unused and defaults to 0.
For DataFrames, specifying
axis=None
will apply the aggregation across both axes.New in version 2.0.0.
skipna (bool, default True) – Exclude NA/null values when computing the result.
numeric_only (bool, default False) – Include only float, int, boolean columns. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Returns:
Examples
>>> s = pd.Series([1, 2, 2, 3], index=['cat', 'dog', 'dog', 'mouse']) >>> s cat 1 dog 2 dog 2 mouse 3 dtype: int64 >>> s.kurt() 1.5
With a DataFrame
>>> df = pd.DataFrame({'a': [1, 2, 2, 3], 'b': [3, 4, 4, 4]}, ... index=['cat', 'dog', 'dog', 'mouse']) >>> df a b cat 1 3 dog 2 4 dog 2 4 mouse 3 4 >>> df.kurt() a 1.5 b 4.0 dtype: float64
With axis=None
>>> df.kurt(axis=None).round(6) -0.988693
Using axis=1
>>> df = pd.DataFrame({'a': [1, 2], 'b': [3, 4], 'c': [3, 4], 'd': [1, 2]}, ... index=['cat', 'dog']) >>> df.kurt(axis=1) cat -6.0 dog -6.0 dtype: float64
- Return type:
Series or scalar
Notes
See pandas API documentation for pandas.DataFrame.kurt for more.
- kurtosis(axis=0, skipna=True, numeric_only=False, **kwargs) Series | float
Return unbiased kurtosis over requested axis.
Kurtosis obtained using Fisher’s definition of kurtosis (kurtosis of normal == 0.0). Normalized by N-1.
- Parameters:
axis ({index (0), columns (1)}) –
Axis for the function to be applied on. For Series this parameter is unused and defaults to 0.
For DataFrames, specifying
axis=None
will apply the aggregation across both axes.New in version 2.0.0.
skipna (bool, default True) – Exclude NA/null values when computing the result.
numeric_only (bool, default False) – Include only float, int, boolean columns. Not implemented for Series.
**kwargs – Additional keyword arguments to be passed to the function.
- Returns:
Examples
>>> s = pd.Series([1, 2, 2, 3], index=['cat', 'dog', 'dog', 'mouse']) >>> s cat 1 dog 2 dog 2 mouse 3 dtype: int64 >>> s.kurt() 1.5
With a DataFrame
>>> df = pd.DataFrame({'a': [1, 2, 2, 3], 'b': [3, 4, 4, 4]}, ... index=['cat', 'dog', 'dog', 'mouse']) >>> df a b cat 1 3 dog 2 4 dog 2 4 mouse 3 4 >>> df.kurt() a 1.5 b 4.0 dtype: float64
With axis=None
>>> df.kurt(axis=None).round(6) -0.988693
Using axis=1
>>> df = pd.DataFrame({'a': [1, 2], 'b': [3, 4], 'c': [3, 4], 'd': [1, 2]}, ... index=['cat', 'dog']) >>> df.kurt(axis=1) cat -6.0 dog -6.0 dtype: float64
- Return type:
Series or scalar
Notes
See pandas API documentation for pandas.DataFrame.kurt for more.
- last(offset) Self
Select final periods of time series data based on a date offset.
Notes
See pandas API documentation for pandas.DataFrame.last, pandas.Series.last for more.
- last_valid_index() int
Return index for last non-NA value or None, if no non-NA value is found.
Notes
See pandas API documentation for pandas.DataFrame.last_valid_index, pandas.Series.last_valid_index for more.
- le(other, axis='columns', level=None) Self
Get less than or equal comparison of BasePandasDataset and other, element-wise (binary operator le).
Notes
See pandas API documentation for pandas.DataFrame.le, pandas.Series.le for more.
- property loc: _LocIndexer
Get a group of rows and columns by label(s) or a boolean array.
Notes
See pandas API documentation for pandas.DataFrame.loc, pandas.Series.loc for more.
- lt(other, axis='columns', level=None) Self
Get less than comparison of BasePandasDataset and other, element-wise (binary operator lt).
Notes
See pandas API documentation for pandas.DataFrame.lt, pandas.Series.lt for more.
- mask(cond, other=_NoDefault.no_default, *, inplace: bool = False, axis: Optional[Axis] = None, level: Optional[Level] = None) Self | None
Replace values where the condition is True.
Notes
See pandas API documentation for pandas.DataFrame.mask, pandas.Series.mask for more.
- max(axis: Axis = 0, skipna=True, numeric_only=False, **kwargs) Series | None
Return the maximum of the values over the requested axis.
Notes
See pandas API documentation for pandas.DataFrame.max, pandas.Series.max for more.
- mean(axis: Axis = 0, skipna=True, numeric_only=False, **kwargs) Series | float
Return the mean of the values over the requested axis.
Notes
See pandas API documentation for pandas.DataFrame.mean, pandas.Series.mean for more.
- median(axis: Axis = 0, skipna=True, numeric_only=False, **kwargs) Series | float
Return the mean of the values over the requested axis.
Notes
See pandas API documentation for pandas.DataFrame.median, pandas.Series.median for more.
- memory_usage(index=True, deep=False) Series | None
Return the memory usage of the BasePandasDataset.
Notes
See pandas API documentation for pandas.DataFrame.memory_usage, pandas.Series.memory_usage for more.
- min(axis: Axis = 0, skipna: bool = True, numeric_only=False, **kwargs) Series | None
Return the minimum of the values over the requested axis.
Notes
See pandas API documentation for pandas.DataFrame.min, pandas.Series.min for more.
- mod(other, axis='columns', level=None, fill_value=None) Self
Get modulo of BasePandasDataset and other, element-wise (binary operator mod).
Notes
See pandas API documentation for pandas.DataFrame.mod, pandas.Series.mod for more.
- mode(axis=0, numeric_only=False, dropna=True) Self
Get the mode(s) of each element along the selected axis.
Notes
See pandas API documentation for pandas.DataFrame.mode, pandas.Series.mode for more.
- modin
alias of
ModinAPI
- move_to(backend: str, inplace: bool = False) Optional[Self]
Move the data in this
BasePandasDataset
from its current backend to the given one.Further operations on this
BasePandasDataset
will use the new backend instead of the current one.- Parameters:
backend (str) – The name of the backend to set.
inplace (bool, default: False) – Whether to modify this
BasePandasDataset
in place.
- Returns:
If
inplace
is False, returns a new instance of theBasePandasDataset
with the given backend. Ifinplace
isTrue
, returns None.- Return type:
BasePandasDataset
or None
Notes
- This method will
convert the data in this
BasePandasDataset
to a pandas DataFrame in this Python processload the data from pandas to the new backend.
Either step may be slow and/or memory-intensive, especially if this
BasePandasDataset
’s data is large, or one or both of the backends do not store their data locally.
- mul(other, axis='columns', level=None, fill_value=None) Self
Get multiplication of BasePandasDataset and other, element-wise (binary operator mul).
Notes
See pandas API documentation for pandas.DataFrame.mul, pandas.Series.mul for more.
- multiply(other, axis='columns', level=None, fill_value=None) Self
Get multiplication of BasePandasDataset and other, element-wise (binary operator mul).
Notes
See pandas API documentation for pandas.DataFrame.multiply, pandas.Series.multiply for more.
- ne(other, axis='columns', level=None) Self
Get Not equal comparison of BasePandasDataset and other, element-wise (binary operator ne).
Notes
See pandas API documentation for pandas.DataFrame.ne, pandas.Series.ne for more.
- notna() Self
Detect existing (non-missing) values.
Notes
See pandas API documentation for pandas.DataFrame.notna, pandas.Series.notna for more.
- notnull() Self
Detect existing (non-missing) values.
Notes
See pandas API documentation for pandas.DataFrame.notnull, pandas.Series.notnull for more.
- nunique(axis=0, dropna=True) Series | int
Return number of unique elements in the BasePandasDataset.
Notes
See pandas API documentation for pandas.DataFrame.nunique, pandas.Series.nunique for more.
- pad(*, axis=None, inplace=False, limit=None, downcast=_NoDefault.no_default) Self | None
Synonym for DataFrame.ffill.
Notes
See pandas API documentation for pandas.DataFrame.pad, pandas.Series.pad for more.
- pct_change(periods=1, fill_method=_NoDefault.no_default, limit=_NoDefault.no_default, freq=None, **kwargs) Self
Percentage change between the current and a prior element.
Notes
See pandas API documentation for pandas.DataFrame.pct_change, pandas.Series.pct_change for more.
- pipe(func: Callable[..., T] | tuple[Callable[..., T], str], *args, **kwargs) T
Apply chainable functions that expect BasePandasDataset.
Notes
See pandas API documentation for pandas.DataFrame.pipe, pandas.Series.pipe for more.
- pop(item) Series | Scalar
Return item and drop from frame. Raise KeyError if not found.
Notes
See pandas API documentation for pandas.DataFrame.pop, pandas.Series.pop for more.
- pow(other, axis='columns', level=None, fill_value=None) Self
Get exponential power of BasePandasDataset and other, element-wise (binary operator pow).
Notes
See pandas API documentation for pandas.DataFrame.pow, pandas.Series.pow for more.
- quantile(q, axis, numeric_only, interpolation, method) DataFrame | Series | Scalar
Return values at the given quantile over requested axis.
Notes
See pandas API documentation for pandas.DataFrame.quantile, pandas.Series.quantile for more.
- radd(other, axis='columns', level=None, fill_value=None) Self
Return addition of BasePandasDataset and other, element-wise (binary operator radd).
Notes
See pandas API documentation for pandas.DataFrame.radd, pandas.Series.radd for more.
- rank(axis=0, method: str = 'average', numeric_only=False, na_option: str = 'keep', ascending: bool = True, pct: bool = False) Self
Compute numerical data ranks (1 through n) along axis.
By default, equal values are assigned a rank that is the average of the ranks of those values.
- Parameters:
axis ({0 or 'index', 1 or 'columns'}, default 0) – Index to direct ranking. For Series this parameter is unused and defaults to 0.
method ({'average', 'min', 'max', 'first', 'dense'}, default 'average') –
How to rank the group of records that have the same value (i.e. ties):
average: average rank of the group
min: lowest rank in the group
max: highest rank in the group
first: ranks assigned in order they appear in the array
dense: like ‘min’, but rank always increases by 1 between groups.
numeric_only (bool, default False) –
For DataFrame objects, rank only numeric columns if set to True.
Changed in version 2.0.0: The default value of
numeric_only
is nowFalse
.na_option ({'keep', 'top', 'bottom'}, default 'keep') –
How to rank NaN values:
keep: assign NaN rank to NaN values
top: assign lowest rank to NaN values
bottom: assign highest rank to NaN values
ascending (bool, default True) – Whether or not the elements should be ranked in ascending order.
pct (bool, default False) – Whether or not to display the returned rankings in percentile form.
- Returns:
Return a Series or DataFrame with data ranks as values.
- Return type:
same type as caller
See also
core.groupby.DataFrameGroupBy.rank
Rank of values within each group.
core.groupby.SeriesGroupBy.rank
Rank of values within each group.
Examples
>>> df = pd.DataFrame(data={'Animal': ['cat', 'penguin', 'dog', ... 'spider', 'snake'], ... 'Number_legs': [4, 2, 4, 8, np.nan]}) >>> df Animal Number_legs 0 cat 4.0 1 penguin 2.0 2 dog 4.0 3 spider 8.0 4 snake NaN
Ties are assigned the mean of the ranks (by default) for the group.
>>> s = pd.Series(range(5), index=list("abcde")) >>> s["d"] = s["b"] >>> s.rank() a 1.0 b 2.5 c 4.0 d 2.5 e 5.0 dtype: float64
The following example shows how the method behaves with the above parameters:
default_rank: this is the default behaviour obtained without using any parameter.
max_rank: setting
method = 'max'
the records that have the same values are ranked using the highest rank (e.g.: since ‘cat’ and ‘dog’ are both in the 2nd and 3rd position, rank 3 is assigned.)NA_bottom: choosing
na_option = 'bottom'
, if there are records with NaN values they are placed at the bottom of the ranking.pct_rank: when setting
pct = True
, the ranking is expressed as percentile rank.
>>> df['default_rank'] = df['Number_legs'].rank() >>> df['max_rank'] = df['Number_legs'].rank(method='max') >>> df['NA_bottom'] = df['Number_legs'].rank(na_option='bottom') >>> df['pct_rank'] = df['Number_legs'].rank(pct=True) >>> df Animal Number_legs default_rank max_rank NA_bottom pct_rank 0 cat 4.0 2.5 3.0 2.5 0.625 1 penguin 2.0 1.0 1.0 1.0 0.250 2 dog 4.0 2.5 3.0 2.5 0.625 3 spider 8.0 4.0 4.0 4.0 1.000 4 snake NaN NaN NaN 5.0 NaN
Notes
See pandas API documentation for pandas.DataFrame.rank for more.
- rdiv(other, axis='columns', level=None, fill_value=None) Self
Get floating division of BasePandasDataset and other, element-wise (binary operator rtruediv).
Notes
See pandas API documentation for pandas.DataFrame.rdiv, pandas.Series.rdiv for more.
- reindex(index=None, columns=None, copy=True, **kwargs) Self
Conform BasePandasDataset to new index with optional filling logic.
Notes
See pandas API documentation for pandas.DataFrame.reindex, pandas.Series.reindex for more.
- rename_axis(mapper=_NoDefault.no_default, *, index=_NoDefault.no_default, columns=_NoDefault.no_default, axis=0, copy=None, inplace=False) DataFrame | Series | None
Set the name of the axis for the index or columns.
Notes
See pandas API documentation for pandas.DataFrame.rename_axis, pandas.Series.rename_axis for more.
- reorder_levels(order, axis=0) Self
Rearrange index levels using input order.
Notes
See pandas API documentation for pandas.DataFrame.reorder_levels, pandas.Series.reorder_levels for more.
- resample(rule, axis: Axis = _NoDefault.no_default, closed: Optional[str] = None, label: Optional[str] = None, convention: str = _NoDefault.no_default, kind: Optional[str] = _NoDefault.no_default, on: Level = None, level: Level = None, origin: str | TimestampConvertibleTypes = 'start_day', offset: Optional[TimedeltaConvertibleTypes] = None, group_keys=False) Resampler
Resample time-series data.
Notes
See pandas API documentation for pandas.DataFrame.resample, pandas.Series.resample for more.
- reset_index(level: IndexLabel = None, *, drop: bool = False, inplace: bool = False, col_level: Hashable = 0, col_fill: Hashable = '', allow_duplicates=_NoDefault.no_default, names: Hashable | Sequence[Hashable] = None) DataFrame | Series | None
Reset the index, or a level of it.
Notes
See pandas API documentation for pandas.DataFrame.reset_index, pandas.Series.reset_index for more.
- rfloordiv(other, axis='columns', level=None, fill_value=None) Self
Get integer division of BasePandasDataset and other, element-wise (binary operator rfloordiv).
Notes
See pandas API documentation for pandas.DataFrame.rfloordiv, pandas.Series.rfloordiv for more.
- rmod(other, axis='columns', level=None, fill_value=None) Self
Get modulo of BasePandasDataset and other, element-wise (binary operator rmod).
Notes
See pandas API documentation for pandas.DataFrame.rmod, pandas.Series.rmod for more.
- rmul(other, axis='columns', level=None, fill_value=None) Self
Get Multiplication of dataframe and other, element-wise (binary operator rmul).
Notes
See pandas API documentation for pandas.DataFrame.rmul, pandas.Series.rmul for more.
- rolling(window, min_periods: int | None = None, center: bool = False, win_type: str | None = None, on: str | None = None, axis: Axis = _NoDefault.no_default, closed: str | None = None, step: int | None = None, method: str = 'single') Rolling | Window
Provide rolling window calculations.
Notes
See pandas API documentation for pandas.DataFrame.rolling, pandas.Series.rolling for more.
- round(decimals=0, *args, **kwargs) Self
Round a BasePandasDataset to a variable number of decimal places.
Notes
See pandas API documentation for pandas.DataFrame.round, pandas.Series.round for more.
- rpow(other, axis='columns', level=None, fill_value=None) Self
Get exponential power of BasePandasDataset and other, element-wise (binary operator rpow).
Notes
See pandas API documentation for pandas.DataFrame.rpow, pandas.Series.rpow for more.
- rsub(other, axis='columns', level=None, fill_value=None) Self
Get subtraction of BasePandasDataset and other, element-wise (binary operator rsub).
Notes
See pandas API documentation for pandas.DataFrame.rsub, pandas.Series.rsub for more.
- rtruediv(other, axis='columns', level=None, fill_value=None) Self
Get floating division of BasePandasDataset and other, element-wise (binary operator rtruediv).
Notes
See pandas API documentation for pandas.DataFrame.rtruediv, pandas.Series.rtruediv for more.
- sample(n: int | None = None, frac: float | None = None, replace: bool = False, weights=None, random_state: RandomState | None = None, axis: Axis | None = None, ignore_index: bool = False) Self
Return a random sample of items from an axis of object.
Notes
See pandas API documentation for pandas.DataFrame.sample, pandas.Series.sample for more.
- sem(axis: Axis = 0, skipna: bool = True, ddof: int = 1, numeric_only=False, **kwargs) Series | float
Return unbiased standard error of the mean over requested axis.
Notes
See pandas API documentation for pandas.DataFrame.sem, pandas.Series.sem for more.
- set_axis(labels, *, axis: Axis = 0, copy=None) Self
Assign desired index to given axis.
Notes
See pandas API documentation for pandas.DataFrame.set_axis, pandas.Series.set_axis for more.
- set_backend(backend: str, inplace: bool = False) Optional[Self]
Move the data in this
BasePandasDataset
from its current backend to the given one.Further operations on this
BasePandasDataset
will use the new backend instead of the current one.- Parameters:
backend (str) – The name of the backend to set.
inplace (bool, default: False) – Whether to modify this
BasePandasDataset
in place.
- Returns:
If
inplace
is False, returns a new instance of theBasePandasDataset
with the given backend. Ifinplace
isTrue
, returns None.- Return type:
BasePandasDataset
or None
Notes
- This method will
convert the data in this
BasePandasDataset
to a pandas DataFrame in this Python processload the data from pandas to the new backend.
Either step may be slow and/or memory-intensive, especially if this
BasePandasDataset
’s data is large, or one or both of the backends do not store their data locally.
- set_flags(*, copy: bool = False, allows_duplicate_labels: Optional[bool] = None) Self
Return a new BasePandasDataset with updated flags.
Notes
See pandas API documentation for pandas.DataFrame.set_flags, pandas.Series.set_flags for more.
- shift(periods: int = 1, freq=None, axis: Axis = 0, fill_value: Hashable = _NoDefault.no_default, suffix=None) Self | DataFrame
Shift index by desired number of periods with an optional time freq.
Notes
See pandas API documentation for pandas.DataFrame.shift, pandas.Series.shift for more.
- property size: int
Return an int representing the number of elements in this BasePandasDataset object.
Notes
See pandas API documentation for pandas.DataFrame.size, pandas.Series.size for more.
- skew(axis: Axis = 0, skipna: bool = True, numeric_only=False, **kwargs) Series | float
Return unbiased skew over requested axis.
Notes
See pandas API documentation for pandas.DataFrame.skew, pandas.Series.skew for more.
- sort_index(*, axis=0, level=None, ascending=True, inplace=False, kind='quicksort', na_position='last', sort_remaining=True, ignore_index: bool = False, key: Optional[IndexKeyFunc] = None) Self | None
Sort object by labels (along an axis).
Notes
See pandas API documentation for pandas.DataFrame.sort_index, pandas.Series.sort_index for more.
- sort_values(by, *, axis=0, ascending=True, inplace: bool = False, kind='quicksort', na_position='last', ignore_index: bool = False, key: Optional[IndexKeyFunc] = None) Self | None
Sort by the values along either axis.
Notes
See pandas API documentation for pandas.DataFrame.sort_values, pandas.Series.sort_values for more.
- std(axis: Axis = 0, skipna: bool = True, ddof: int = 1, numeric_only=False, **kwargs) Series | float
Return sample standard deviation over requested axis.
Notes
See pandas API documentation for pandas.DataFrame.std, pandas.Series.std for more.
- sub(other, axis='columns', level=None, fill_value=None) Self
Get subtraction of BasePandasDataset and other, element-wise (binary operator sub).
Notes
See pandas API documentation for pandas.DataFrame.sub, pandas.Series.sub for more.
- subtract(other, axis='columns', level=None, fill_value=None) Self
Get subtraction of BasePandasDataset and other, element-wise (binary operator sub).
Notes
See pandas API documentation for pandas.DataFrame.subtract, pandas.Series.subtract for more.
- swapaxes(axis1, axis2, copy=None) Self
Interchange axes and swap values axes appropriately.
Notes
See pandas API documentation for pandas.DataFrame.swapaxes, pandas.Series.swapaxes for more.
- swaplevel(i=-2, j=-1, axis=0) Self
Swap levels i and j in a MultiIndex.
Notes
See pandas API documentation for pandas.DataFrame.swaplevel, pandas.Series.swaplevel for more.
- tail(n=5) Self
Return the last n rows.
Notes
See pandas API documentation for pandas.DataFrame.tail, pandas.Series.tail for more.
- take(indices, axis=0, **kwargs) Self
Return the elements in the given positional indices along an axis.
Notes
See pandas API documentation for pandas.DataFrame.take, pandas.Series.take for more.
- to_clipboard(excel=True, sep=None, **kwargs)
Copy object to the system clipboard.
Notes
See pandas API documentation for pandas.DataFrame.to_clipboard, pandas.Series.to_clipboard for more.
- to_csv(path_or_buf=None, sep=',', na_rep='', float_format=None, columns=None, header=True, index=True, index_label=None, mode='w', encoding=None, compression='infer', quoting=None, quotechar='"', lineterminator=None, chunksize=None, date_format=None, doublequote=True, escapechar=None, decimal='.', errors: str = 'strict', storage_options: StorageOptions = None) str | None
Write object to a comma-separated values (csv) file.
- Parameters:
path_or_buf (str, path object, file-like object, or None, default None) – String, path object (implementing os.PathLike[str]), or file-like object implementing a write() function. If None, the result is returned as a string. If a non-binary file object is passed, it should be opened with newline=’’, disabling universal newlines. If a binary file object is passed, mode might need to contain a ‘b’.
sep (str, default ',') – String of length 1. Field delimiter for the output file.
na_rep (str, default '') – Missing data representation.
float_format (str, Callable, default None) – Format string for floating point numbers. If a Callable is given, it takes precedence over other numeric formatting parameters, like decimal.
columns (sequence, optional) – Columns to write.
header (bool or list of str, default True) – Write out the column names. If a list of strings is given it is assumed to be aliases for the column names.
index (bool, default True) – Write row names (index).
index_label (str or sequence, or False, default None) – Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the object uses MultiIndex. If False do not print fields for index names. Use index_label=False for easier importing in R.
mode ({'w', 'x', 'a'}, default 'w') –
Forwarded to either open(mode=) or fsspec.open(mode=) to control the file opening. Typical values include:
’w’, truncate the file first.
’x’, exclusive creation, failing if the file already exists.
’a’, append to the end of file if it exists.
encoding (str, optional) – A string representing the encoding to use in the output file, defaults to ‘utf-8’. encoding is not supported if path_or_buf is a non-binary file object.
compression (str or dict, default 'infer') –
For on-the-fly compression of the output data. If ‘infer’ and ‘path_or_buf’ is path-like, then detect compression from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’, ‘.xz’, ‘.zst’, ‘.tar’, ‘.tar.gz’, ‘.tar.xz’ or ‘.tar.bz2’ (otherwise no compression). Set to
None
for no compression. Can also be a dict with key'method'
set to one of {'zip'
,'gzip'
,'bz2'
,'zstd'
,'xz'
,'tar'
} and other key-value pairs are forwarded tozipfile.ZipFile
,gzip.GzipFile
,bz2.BZ2File
,zstandard.ZstdCompressor
,lzma.LZMAFile
ortarfile.TarFile
, respectively. As an example, the following could be passed for faster compression and to create a reproducible gzip archive:compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}
.New in version 1.5.0: Added support for .tar files.
May be a dict with key ‘method’ as compression mode and other entries as additional compression options if compression mode is ‘zip’.
Passing compression options as keys in dict is supported for compression modes ‘gzip’, ‘bz2’, ‘zstd’, and ‘zip’.
quoting (optional constant from csv module) – Defaults to csv.QUOTE_MINIMAL. If you have set a float_format then floats are converted to strings and thus csv.QUOTE_NONNUMERIC will treat them as non-numeric.
quotechar (str, default '"') – String of length 1. Character used to quote fields.
lineterminator (str, optional) –
The newline character or character sequence to use in the output file. Defaults to os.linesep, which depends on the OS in which this method is called (’\n’ for linux, ‘\r\n’ for Windows, i.e.).
Changed in version 1.5.0: Previously was line_terminator, changed for consistency with read_csv and the standard library ‘csv’ module.
chunksize (int or None) – Rows to write at a time.
date_format (str, default None) – Format string for datetime objects.
doublequote (bool, default True) – Control quoting of quotechar inside a field.
escapechar (str, default None) – String of length 1. Character used to escape sep and quotechar when appropriate.
decimal (str, default '.') – Character recognized as decimal separator. E.g. use ‘,’ for European data.
errors (str, default 'strict') – Specifies how encoding and decoding errors are to be handled. See the errors argument for
open()
for a full list of options.storage_options (dict, optional) – Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to
urllib.request.Request
as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded tofsspec.open
. Please seefsspec
andurllib
for more details, and for more examples on storage options refer here.
- Returns:
If path_or_buf is None, returns the resulting csv format as a string. Otherwise returns None.
- Return type:
None or str
See also
read_csv
Load a CSV file into a DataFrame.
to_excel
Write DataFrame to an Excel file.
Examples
Create ‘out.csv’ containing ‘df’ without indices
>>> df = pd.DataFrame({'name': ['Raphael', 'Donatello'], ... 'mask': ['red', 'purple'], ... 'weapon': ['sai', 'bo staff']}) >>> df.to_csv('out.csv', index=False)
Create ‘out.zip’ containing ‘out.csv’
>>> df.to_csv(index=False) 'name,mask,weapon\nRaphael,red,sai\nDonatello,purple,bo staff\n' >>> compression_opts = dict(method='zip', ... archive_name='out.csv') >>> df.to_csv('out.zip', index=False, ... compression=compression_opts)
To write a csv file to a new folder or nested folder you will first need to create it using either Pathlib or os:
>>> from pathlib import Path >>> filepath = Path('folder/subfolder/out.csv') >>> filepath.parent.mkdir(parents=True, exist_ok=True) >>> df.to_csv(filepath)
>>> import os >>> os.makedirs('folder/subfolder', exist_ok=True) >>> df.to_csv('folder/subfolder/out.csv')
Notes
See pandas API documentation for pandas.DataFrame.to_csv, pandas.Series.to_csv for more.
- to_dict(orient='dict', into=<class 'dict'>, index=True) dict
Convert the DataFrame to a dictionary.
The type of the key-value pairs can be customized with the parameters (see below).
- Parameters:
orient (str {'dict', 'list', 'series', 'split', 'tight', 'records', 'index'}) –
Determines the type of the values of the dictionary.
’dict’ (default) : dict like {column -> {index -> value}}
’list’ : dict like {column -> [values]}
’series’ : dict like {column -> Series(values)}
’split’ : dict like {‘index’ -> [index], ‘columns’ -> [columns], ‘data’ -> [values]}
’tight’ : dict like {‘index’ -> [index], ‘columns’ -> [columns], ‘data’ -> [values], ‘index_names’ -> [index.names], ‘column_names’ -> [column.names]}
’records’ : list like [{column -> value}, … , {column -> value}]
’index’ : dict like {index -> {column -> value}}
New in version 1.4.0: ‘tight’ as an allowed value for the
orient
argumentinto (class, default dict) – The collections.abc.MutableMapping subclass used for all Mappings in the return value. Can be the actual class or an empty instance of the mapping type you want. If you want a collections.defaultdict, you must pass it initialized.
index (bool, default True) –
Whether to include the index item (and index_names item if orient is ‘tight’) in the returned dictionary. Can only be
False
when orient is ‘split’ or ‘tight’.New in version 2.0.0.
- Returns:
Return a collections.abc.MutableMapping object representing the DataFrame. The resulting transformation depends on the orient parameter.
- Return type:
dict, list or collections.abc.MutableMapping
See also
DataFrame.from_dict
Create a DataFrame from a dictionary.
DataFrame.to_json
Convert a DataFrame to JSON format.
Examples
>>> df = pd.DataFrame({'col1': [1, 2], ... 'col2': [0.5, 0.75]}, ... index=['row1', 'row2']) >>> df col1 col2 row1 1 0.50 row2 2 0.75 >>> df.to_dict() {'col1': {'row1': 1, 'row2': 2}, 'col2': {'row1': 0.5, 'row2': 0.75}}
You can specify the return orientation.
>>> df.to_dict('series') {'col1': row1 1 row2 2 Name: col1, dtype: int64, 'col2': row1 0.50 row2 0.75 Name: col2, dtype: float64}
>>> df.to_dict('split') {'index': ['row1', 'row2'], 'columns': ['col1', 'col2'], 'data': [[1, 0.5], [2, 0.75]]}
>>> df.to_dict('records') [{'col1': 1, 'col2': 0.5}, {'col1': 2, 'col2': 0.75}]
>>> df.to_dict('index') {'row1': {'col1': 1, 'col2': 0.5}, 'row2': {'col1': 2, 'col2': 0.75}}
>>> df.to_dict('tight') {'index': ['row1', 'row2'], 'columns': ['col1', 'col2'], 'data': [[1, 0.5], [2, 0.75]], 'index_names': [None], 'column_names': [None]}
You can also specify the mapping type.
>>> from collections import OrderedDict, defaultdict >>> df.to_dict(into=OrderedDict) OrderedDict([('col1', OrderedDict([('row1', 1), ('row2', 2)])), ('col2', OrderedDict([('row1', 0.5), ('row2', 0.75)]))])
If you want a defaultdict, you need to initialize it:
>>> dd = defaultdict(list) >>> df.to_dict('records', into=dd) [defaultdict(<class 'list'>, {'col1': 1, 'col2': 0.5}), defaultdict(<class 'list'>, {'col1': 2, 'col2': 0.75})]
Notes
See pandas API documentation for pandas.DataFrame.to_dict, pandas.Series.to_dict for more.
- to_excel(excel_writer, sheet_name='Sheet1', na_rep='', float_format=None, columns=None, header=True, index=True, index_label=None, startrow=0, startcol=0, engine=None, merge_cells=True, inf_rep='inf', freeze_panes=None, storage_options: Optional[dict[str, Any]] = None, engine_kwargs=None) None
Write object to an Excel sheet.
Notes
See pandas API documentation for pandas.DataFrame.to_excel, pandas.Series.to_excel for more.
- to_hdf(path_or_buf, key: str, mode: Literal['a', 'w', 'r+'] = 'a', complevel: int | None = None, complib: Literal['zlib', 'lzo', 'bzip2', 'blosc'] | None = None, append: bool = False, format: Literal['fixed', 'table'] | None = None, index: bool = True, min_itemsize: int | dict[str, int] | None = None, nan_rep=None, dropna: bool | None = None, data_columns: Literal[True] | list[str] | None = None, errors: str = 'strict', encoding: str = 'UTF-8') None
Write the contained data to an HDF5 file using HDFStore.
Notes
See pandas API documentation for pandas.DataFrame.to_hdf, pandas.Series.to_hdf for more.
- to_json(path_or_buf=None, orient=None, date_format=None, double_precision=10, force_ascii=True, date_unit='ms', default_handler=None, lines=False, compression='infer', index=None, indent=None, storage_options: StorageOptions = None, mode='w') str | None
Convert the object to a JSON string.
Notes
See pandas API documentation for pandas.DataFrame.to_json, pandas.Series.to_json for more.
- to_latex(buf=None, columns=None, header=True, index=True, na_rep='NaN', formatters=None, float_format=None, sparsify=None, index_names=True, bold_rows=False, column_format=None, longtable=None, escape=None, encoding=None, decimal='.', multicolumn=None, multicolumn_format=None, multirow=None, caption=None, label=None, position=None) str | None
Render object to a LaTeX tabular, longtable, or nested table.
Notes
See pandas API documentation for pandas.DataFrame.to_latex, pandas.Series.to_latex for more.
- to_markdown(buf=None, mode: str = 'wt', index: bool = True, storage_options: Optional[dict[str, Any]] = None, **kwargs) str
Print BasePandasDataset in Markdown-friendly format.
Notes
See pandas API documentation for pandas.DataFrame.to_markdown, pandas.Series.to_markdown for more.
- to_numpy(dtype=None, copy=False, na_value=_NoDefault.no_default) ndarray
Convert the BasePandasDataset to a NumPy array or a Modin wrapper for NumPy array.
Notes
See pandas API documentation for pandas.DataFrame.to_numpy, pandas.Series.to_numpy for more.
- to_period(freq=None, axis=0, copy=None) Self
Convert BasePandasDataset from DatetimeIndex to PeriodIndex.
Notes
See pandas API documentation for pandas.DataFrame.to_period, pandas.Series.to_period for more.
- to_pickle(path, compression: Optional[Union[Literal['infer', 'gzip', 'bz2', 'zip', 'xz', 'zstd', 'tar'], dict[str, Any]]] = 'infer', protocol: int = 5, storage_options: Optional[dict[str, Any]] = None) None
Pickle (serialize) object to file.
Notes
See pandas API documentation for pandas.DataFrame.to_pickle, pandas.Series.to_pickle for more.
- to_sql(name, con, schema=None, if_exists='fail', index=True, index_label=None, chunksize=None, dtype=None, method=None) int | None
Write records stored in a BasePandasDataset to a SQL database.
Notes
See pandas API documentation for pandas.DataFrame.to_sql, pandas.Series.to_sql for more.
- to_string(buf=None, columns=None, col_space=None, header=True, index=True, na_rep='NaN', formatters=None, float_format=None, sparsify=None, index_names=True, justify=None, max_rows=None, min_rows=None, max_cols=None, show_dimensions=False, decimal='.', line_width=None, max_colwidth=None, encoding=None) str | None
Render a BasePandasDataset to a console-friendly tabular output.
Notes
See pandas API documentation for pandas.DataFrame.to_string, pandas.Series.to_string for more.
- to_timestamp(freq=None, how='start', axis=0, copy=None) Self
Cast to DatetimeIndex of timestamps, at beginning of period.
Notes
See pandas API documentation for pandas.DataFrame.to_timestamp, pandas.Series.to_timestamp for more.
- to_xarray()
Return an xarray object from the BasePandasDataset.
Notes
See pandas API documentation for pandas.DataFrame.to_xarray, pandas.Series.to_xarray for more.
- transform(func, axis=0, *args, **kwargs) Self
Call
func
on self producing a BasePandasDataset with the same axis shape as self.Notes
See pandas API documentation for pandas.DataFrame.transform, pandas.Series.transform for more.
- truediv(other, axis='columns', level=None, fill_value=None) Self
Get floating division of BasePandasDataset and other, element-wise (binary operator truediv).
Notes
See pandas API documentation for pandas.DataFrame.truediv, pandas.Series.truediv for more.
- truncate(before=None, after=None, axis=None, copy=None) Self
Truncate a BasePandasDataset before and after some index value.
Notes
See pandas API documentation for pandas.DataFrame.truncate, pandas.Series.truncate for more.
- tz_convert(tz, axis=0, level=None, copy=None) Self
Convert tz-aware axis to target time zone.
Notes
See pandas API documentation for pandas.DataFrame.tz_convert, pandas.Series.tz_convert for more.
- tz_localize(tz, axis=0, level=None, copy=None, ambiguous='raise', nonexistent='raise') Self
Localize tz-naive index of a BasePandasDataset to target time zone.
Notes
See pandas API documentation for pandas.DataFrame.tz_localize, pandas.Series.tz_localize for more.
- value_counts(subset: Sequence[Hashable] | None = None, normalize: bool = False, sort: bool = True, ascending: bool = False, dropna: bool = True) Series
Return a Series containing the frequency of each distinct row in the Dataframe.
- Parameters:
subset (label or list of labels, optional) – Columns to use when counting unique combinations.
normalize (bool, default False) – Return proportions rather than frequencies.
sort (bool, default True) – Sort by frequencies when True. Sort by DataFrame column values when False.
ascending (bool, default False) – Sort in ascending order.
dropna (bool, default True) –
Don’t include counts of rows that contain NA values.
New in version 1.3.0.
- Return type:
See also
Series.value_counts
Equivalent method on Series.
Notes
See pandas API documentation for pandas.DataFrame.value_counts for more. The returned Series will have a MultiIndex with one level per input column but an Index (non-multi) for a single label. By default, rows that contain any NA values are omitted from the result. By default, the resulting Series will be in descending order so that the first element is the most frequently-occurring row.
Examples
>>> df = pd.DataFrame({'num_legs': [2, 4, 4, 6], ... 'num_wings': [2, 0, 0, 0]}, ... index=['falcon', 'dog', 'cat', 'ant']) >>> df num_legs num_wings falcon 2 2 dog 4 0 cat 4 0 ant 6 0
>>> df.value_counts() num_legs num_wings 4 0 2 2 2 1 6 0 1 Name: count, dtype: int64
>>> df.value_counts(sort=False) num_legs num_wings 2 2 1 4 0 2 6 0 1 Name: count, dtype: int64
>>> df.value_counts(ascending=True) num_legs num_wings 2 2 1 6 0 1 4 0 2 Name: count, dtype: int64
>>> df.value_counts(normalize=True) num_legs num_wings 4 0 0.50 2 2 0.25 6 0 0.25 Name: proportion, dtype: float64
With dropna set to False we can also count rows with NA values.
>>> df = pd.DataFrame({'first_name': ['John', 'Anne', 'John', 'Beth'], ... 'middle_name': ['Smith', pd.NA, pd.NA, 'Louise']}) >>> df first_name middle_name 0 John Smith 1 Anne <NA> 2 John <NA> 3 Beth Louise
>>> df.value_counts() first_name middle_name Beth Louise 1 John Smith 1 Name: count, dtype: int64
>>> df.value_counts(dropna=False) first_name middle_name Anne NaN 1 Beth Louise 1 John Smith 1 NaN 1 Name: count, dtype: int64
>>> df.value_counts("first_name") first_name John 2 Anne 1 Beth 1 Name: count, dtype: int64
- property values: ndarray
Return a NumPy representation of the BasePandasDataset.
Notes
See pandas API documentation for pandas.DataFrame.values, pandas.Series.values for more.
- var(axis: Axis = 0, skipna: bool = True, ddof: int = 1, numeric_only=False, **kwargs) Series | float
Return unbiased variance over requested axis.
Notes
See pandas API documentation for pandas.DataFrame.var, pandas.Series.var for more.
- xs(key, axis=0, level=None, drop_level: bool = True) Self
Return cross-section from the Series/DataFrame.
Notes
See pandas API documentation for pandas.DataFrame.xs, pandas.Series.xs for more.