Aggregations Module¶

This module is inspired by the exercises in the book “Advances in Financial Machine Learning” by Marcos Lopez de Prado.

Data Aggregation.

Functions:

`generate_count_grouper_column`(data, ...)	It adds a column with a value for each rows count.
`ohlc_group`(data, column_to_ohlc, group_column)	Creates OHLC columns for a column based on group by other column with discrete values.
`sum_split_by_boolean_column_and_group`(data)	Splits the sum of a numeric column into two new columns based on a boolean column and groups the data using another column.
`count_trues_cumulative`(data, bool_column, ...)	Add a new column to the DataFrame with cumulative numbers for each True value in a boolean column.
`drop_aggregated`(data, group_column[, by])	Drop lines except the first/last/min/max etc row of each group_column streak.
`tag_by_accumulation`(trades, threshold[, ...])	Creates integer sequence by column value threshold accumulation.
`resample_klines`(data, tick_interval)	Resamples a DataFrame of klines to a different frequency.
`oversample`(data, new_interval)	Expands a DataFrame of klines by calculating aggregated values for new columns from the start of the oversampled interval to each row.
`time_index_from_timestamps`(data[, ...])	Assumes existing timestamp column or at least Open timestamp column.
`columns_restriction`(data, mode[, extra])	Filter columns by preset.
`generate_volume_column`(data, add_cols[, ...])	Add two columns to generate volume of base.
`sign_of_price`(data[, col_name])	Creates a new dataframe with a column with the sign of the price by each trade.
`tick_bars`(trades, size)	Creates Tick Bars OHLC bars from trades-
`volume_bars`(trades, threshold)	Creates Volume Bars OHLC bars from trades.
`dollar_bars`(trades, threshold)	Creates Dollar (or quote) Bars OHLC bars from trades.
`ohlc_bars`(rows_with_bar_counter, index[, mode])	Generate an OHLC dataframe from a trades dataframe with a column containing sampled numeric tags for each bar's aggregation.
`imbalance_bars_divergent`(trades, ...)	Generates candles by grouping each accumulated imbalance defined by:
`imbalance_bars_fixed`(trades, imbalance)	Generates candles by grouping each accumulated fixed imbalance threshold defined by:
`tick_imbalance_bars`(trades[, window])

Classes:

ImbalanceBars(trades, bar_type[, method, ...])

Sample imbalance bars.

generate_count_grouper_column(data: DataFrame, grouper_name: str, size: int) → DataFrame[source]¶

It adds a column with a value for each rows count.

Parameters:

data (pd.Dataframe) – A dataframe with at least two columns.
grouper_name (str) – Name for the column to be used as grouper.
size (int) – Count of rows to declare grouper integer sequence.

Returns:

A copy of the dataframe with a new column.

ohlc_group(data: DataFrame, column_to_ohlc: str, group_column: str) → DataFrame[source]¶

Creates OHLC columns for a column based on group by other column with discrete values.

Parameters:

data (pd.Dataframe) – A dataframe with at least two columns.
column_to_ohlc (str) – Column to sparse values to OHLC columns in each group.
group_column (str) – This column will be the grouping key.

Returns:

A copy of the dataframe with OHLC data.

sum_split_by_boolean_column_and_group(data: DataFrame, column_to_split_sum: str = 'Quantity', bool_col: str = 'Buyer was maker', group_column: str = 'group') → DataFrame[source]¶

Splits the sum of a numeric column into two new columns based on a boolean column and groups the data using another column.

This function is useful for generating two new columns, one for the true values and another for the false values, with the sum of the specified numeric column for each group and filling the remaining cells with NaNs.

Parameters:

data (pd.DataFrame) – A dataframe with at least 3 columns.
column_to_split_sum (str, optional) – Numeric column with data to sum by group and split by boolean column, defaults to “Quantity”.
bool_col (str, optional) – Column to define splitting, defaults to “Buyer was maker”.
group_column (str, optional) – This column will be the grouping key, defaults to “group”.

Returns:

A copy of the dataframe with splitted sum by group from the column to split sum.

Return type:

pd.DataFrame

count_trues_cumulative(data: DataFrame, bool_column: str, new_column: str) → DataFrame[source]¶

Add a new column to the DataFrame with cumulative numbers for each True value in a boolean column.

Parameters:

data (pd.DataFrame) – The input DataFrame.
bool_column (str) – The name of the boolean column to evaluate.
new_column (str) – The name of the new column to store the cumulative numbers for True values.

Returns:

A new DataFrame with an additional column containing the cumulative numbers for True values.

Return type:

pd.DataFrame

drop_aggregated(data: DataFrame, group_column: str, by='last') → DataFrame[source]¶

Drop lines except the first/last/min/max etc row of each group_column streak.

It assumes that the grouping column is a serie of integers to group it by blocks.

Parameters:

data (pd.Dataframe) – A dataframe.
group_column (str) – This column will be the grouping key.
by (str) – Pandas valid aggregation method.

Returns:

A copy of the dataframe with just the first row each grouper streak.

tag_by_accumulation(trades: DataFrame, threshold: float, agg_column: str = 'Quantity', grouper_name: str = 'group') → DataFrame[source]¶

Creates integer sequence by column value threshold accumulation.

Parameters:

trades (pd.DataFrame) – Expected binpan aggregated trades or atomic trades dataframe.
agg_column (str) – Name of the column to group by volume accumulation.
grouper_name (str) – Name for the column to be used as grouper.
threshold (float) – Size of volume aggregated in bars to be compiled.

Returns:

resample_klines(data: DataFrame, tick_interval: str) → DataFrame[source]¶

Resamples a DataFrame of klines to a different frequency.

Parameters:

data (pd.DataFrame) – The original DataFrame of klines. The index must be a DatetimeIndex.
tick_interval (str) – The new frequency for the klines. This can be any Binance frequency.

Returns:

A new DataFrame with the resampled klines.

oversample(data: DataFrame, new_interval: str) → DataFrame[source]¶

Expands a DataFrame of klines by calculating aggregated values for new columns from the start of the oversampled interval to each row.

Parameters:

data (pd.DataFrame) – The original DataFrame of klines. The index must be a DatetimeIndex.
new_interval (str) – The new frequency for oversampling. This can be any Binance frequency.

Returns:

An expanded DataFrame with oversampled data.

time_index_from_timestamps(data: DataFrame, index_name: str = None, timezone: str = 'Europe/Madrid', drop_col: bool = False)[source]¶

Assumes existing timestamp column or at least Open timestamp column.

Parameters:

data (pd.Dataframe) – A dataframe.
drop_col (bool) – Drop column applied as index.
index_name (str) – Name for the resulting index.
timezone (str) – The index of the pandas dataframe in the object can be converted to any timezone, i.e. “Europe/Madrid” - TZ database: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

Return pd.DataFrame:

A dataframe copy with the new index. Timestamp columns will be not dropped.

columns_restriction(data: DataFrame, mode: str, extra=None) → DataFrame[source]¶

Filter columns by preset.

Parameters:

data (pd.DataFrame) – A dataframe.
mode (str) – Presets are: TB, VB, DB, VIB, DIB, TRB, VRB, DRB. All from chapter 2 of AFML book (Marcos López de Prado). https://www.amazon.com/-/es/Marcos-Lopez-Prado/dp/1119482089
extra (list) – Optional extra columns.

Return pd.DataFrame:

Just preset columns.

generate_volume_column(data: DataFrame, add_cols: tuple, quote_column: bool = None) → DataFrame[source]¶

Add two columns to generate volume of base.

Parameters:

data (pd.DataFrame) – A dataframe.
add_cols (tuple) – Ordered tuple with two columns to add for calculating the total volume.
quote_column (bool) – If True, Volume for quote will be added obtained from Volume and Close price.

Return pd.DataFrame:

A copy with volume added.

sign_of_price(data: DataFrame, col_name: str = 'sign') → DataFrame[source]¶

Creates a new dataframe with a column with the sign of the price by each trade. 1 if the trade increased the price and -1 for decreased price. Also called “tick rule”.

Nans will be filled as zero.

\[\begin{split}b_t= \begin{cases}b_{t-1} & \text { if } \Delta p_t=0 \\ \frac{\left|\Delta p_t\right|}{\Delta p_t} & \text { if } \Delta p_t \neq 0\end{cases}\end{split}\]

Parameters:

data (pd.DataFrame) – Trades dataframe.
col_name (str) – New column name.

Return pd.DataFrame:

A copy of the data with the new column.

tick_bars(trades: DataFrame, size: int) → DataFrame[source]¶

Creates Tick Bars OHLC bars from trades-

Parameters:

trades (pd.DataFrame) – Expected binpan aggregated trades or atomic trades dataframe.
size (int) – Size of ticks in bars to be compiled.

Returns:

A dataframe sampled with the new bars sampling.

volume_bars(trades: DataFrame, threshold: int) → DataFrame[source]¶

Creates Volume Bars OHLC bars from trades.

Parameters:

trades (pd.DataFrame) – Expected binpan aggregated trades or atomic trades dataframe.
threshold (int) – Size of volume threshold in bars to be compiled.

Returns:

A dataframe sampled with the new bars sampling.

dollar_bars(trades: DataFrame, threshold: float) → DataFrame[source]¶

Creates Dollar (or quote) Bars OHLC bars from trades.

Parameters:

trades (pd.DataFrame) – Expected binpan aggregated trades or atomic trades dataframe.
threshold (float) – Size of Dollar (or quote) threshold in bars to be compiled.

Returns:

A dataframe sampled with the new bars sampling.

ohlc_bars(rows_with_bar_counter: list, index: Index, mode: str = 'IB')[source]¶

Generate an OHLC dataframe from a trades dataframe with a column containing sampled numeric tags for each bar’s aggregation.

Parameters:

rows_with_bar_counter (list) – A list of trades with a column containing bar numbers to group into OHLC bars.
index (pd.Index) – Index for the resulting dataframe.
mode (str) – Presets are: TB, VB, DB, VIB, DIB, TRB, VRB, DRB. All from chapter 2 of AFML book (Marcos López de Prado). https://www.amazon.com/-/es/Marcos-Lopez-Prado/dp/1119482089

Returns:

A dataframe with OHLC bars.

class ImbalanceBars(trades: DataFrame, bar_type: str, method: str = 'ema', fixed_imbalance_threshold: int = 1000, window: int = 21, boot_trades: int = 1000, verbose: bool = False, adjust_threshold: bool = False)[source]¶

Bases: object

Sample imbalance bars.

Parameters:

trades (pd.DataFrame) – A dataframe with trades, atomic or aggregated.
bar_type (str) – Can be ‘imbalance’, ‘volume’, ‘dollar’.
method (str) – Expected imbalance calculation method can be ‘fix’, ‘sma’, ‘ema’
fixed_imbalance_threshold (int) – A threshold for fixed imbalance
window (int) – A rolling window size for moving averages.
boot_trades (int) – Number of trades to use for initial imbalance calculation.
verbose (bool) – Prints data while computing bars.
adjust_threshold (bool) – If true, threshold will be multiplied by first price. Ca be useful with volume bars.

Methods:

`initialize_startup_variables`()	Initialize startup variables based on the chosen method.
`get_expected_size`()	Calculate the expected size of the imbalance bars using the chosen method.
`get_expected_probability`()	Calculate the expected probability of the imbalance bars using the chosen method.
`get_expected_imbalance`()	Calculate the expected imbalance value based on the bar type and chosen method.
`sampling_loop`()	Main sampling loop to iterate through trades and create imbalance bars.
`decide_sampling`(my_molecule_sign, ...)	Decide whether to sample the current bar and update variables accordingly.
`construct_bars`()	Construct the resulting imbalance bars DataFrame.

initialize_startup_variables()[source]¶: Initialize startup variables based on the chosen method.

get_expected_size() → float | ndarray[source]¶: Calculate the expected size of the imbalance bars using the chosen method.

get_expected_probability() → float | ndarray[source]¶: Calculate the expected probability of the imbalance bars using the chosen method.

get_expected_imbalance() → float | int[source]¶: Calculate the expected imbalance value based on the bar type and chosen method.

sampling_loop() → None[source]¶: Main sampling loop to iterate through trades and create imbalance bars.

decide_sampling(my_molecule_sign: list, my_molecule_cum_size: int) → tuple[list, int][source]¶

Decide whether to sample the current bar and update variables accordingly.

Sample surpassing data to the next bar.

construct_bars() → None[source]¶: Construct the resulting imbalance bars DataFrame.

imbalance_bars_divergent(trades: DataFrame, starting_imbalance: float) → DataFrame[source]¶

Generates candles by grouping each accumulated imbalance defined by:

\[\theta_T=\sum_{t=1}^T b_t\]

To close a bar, the imbalance must meet expected imbalance while iterating trades, in other words, when expected ticks times the difference between probability of positive signs versus negative signes meets the imbalance.

\[T^*=\underset{T}{\arg \min }\left\{\left|\theta_T\right| \geq \mathrm{E}_0[T]\left|2 \mathrm{P}\left[b_t=1\right]-1\right|\right\}\]

Note

No matter how you obtain expected ticks size and probability for the next imbalance threshold. It explodes. I will focus on fixed threshold in other function.

Parameters:

trades (pd.DataFrame) – Expected binpan aggregated trades or atomic trades dataframe.
starting_imbalance (float) – Starting value for following bars. Its recommended to wait some bars quantity to consider established sizes.

Returns:

A dataframe sampled with the new bars sampling.

imbalance_bars_fixed(trades: DataFrame, imbalance: float) → DataFrame[source]¶

Generates candles by grouping each accumulated fixed imbalance threshold defined by:

\[\theta_T=\sum_{t=1}^T b_t\]

To close a bar, the imbalance must meet expected fixed imbalance while iterating trades.

\[T^*=\underset{T}{\arg \min }\left\{\left|\theta_T\right| \geq \mathrm{E}_0[T]\left|2 \mathrm{P}\left[b_t=1\right]-1\right|\right\}\]

Note

Fixed threshold for rendering imbalance bars.

Parameters:

trades (pd.DataFrame) – Expected binpan aggregated trades or atomic trades dataframe.
imbalance (float) – Fixed value for imbalance threshold.

Returns:

A dataframe sampled with the new bars sampling.

tick_imbalance_bars(trades: DataFrame, window: int = 10)[source]¶