Files Module

This module manages file operations: CSV read/write and file selection. All credentials (Binance, PostgreSQL, …) are managed by panzer (~/.panzer_creds); see Secrets Module.

Functions:

create_dir(my_path)	Create a folder at the designated path if it doesn't exist.
save_dataframe_to_csv(filename, data[, ...])	Save a dataframe in a csv with a separator between columns.
append_dataframe_to_csv(filename, data[, ...])	Add lines to a csv with separator to choose, a dataframe.
find_csvs_in_path([files_path, extension, ...])	Locate all files with desired extension.
move_old_csvs([files_path, extension])	Move all existing files in a path with desired extension to a folder called "old".
extract_filename_metadata(filename, ...[, ...])	Extract metadata from a filename.
read_csv_to_dataframe(filename[, col_sep, ...])	Creates a csv file from a dataframe.
read_file(filename)	Read a file to a list of strings each line.
save_file(filename, data[, mode])	Save a new file from a list of lists each line.
select_file([path, extension, symbol, ...])	Selects from files in the path with the extension passed.
get_database_password()	Get the PostgreSQL password in plain text from panzer (encrypted on disk).
add_any_key(key, key_name)	Store any credential in panzer's CredentialManager (~/.panzer_creds).

create_dir(my_path: str)

Create a folder at the designated path if it doesn’t exist. No error if it already exists

Parameters:

my_path (str) – Path for the new directory.

save_dataframe_to_csv(filename: str, data: DataFrame, col_sep=',', index=False, timestamp=True) → None

Save a dataframe in a csv with a separator between columns. Each field of each column will be enclosed in double quotes.

Parameters:

• filename (str) – The file name.

• data (pd.DataFrame) – The dataframe with date in columns.

• col_sep (str) – Default is ‘,’

• index (bool) – Keep index or not when saving. DEfault is False (drop index)

• timestamp (bool) – Add a timestamp to the file name automatically. Default is true.

append_dataframe_to_csv(filename: str, data: DataFrame, col_sep: str = ',', index: bool = False, header=True) → None

Add lines to a csv with separator to choose, a dataframe. Each field of each column will be enclosed in quotes

Parameters:

• filename (str) – The file to use.

• data (pd.DataFrame) – Data to append to the file.

• col_sep (str) – Expected column separators. Default is “,”

• index (bool) – Keeps index in file as first column/s.

• header (bool) – Keeps header. Default is True.

find_csvs_in_path(files_path: str = '.', extension='csv', name_filter: str = None)

Locate all files with desired extension.

Parameters:

• files_path (str) – A path.

• extension (str) – A extension like “csv”.

• name_filter (str) – A filter to apply to files. Example: “klines” or “aggTrades” or “atomicTrades”.

Returns:

move_old_csvs(files_path: str = '.', extension='csv')

Move all existing files in a path with desired extension to a folder called “old”.

Parameters:

• files_path (str) – A path.

• extension (str) – A extension. Default is “csv”.

extract_filename_metadata(filename: str, expected_data_type: str, expected_symbol: str = None, expected_interval: str = None, expected_timezone: str = None) → tuple

Extract metadata from a filename. Symbol, tick interval, timezone and start timestamp and end timestamp. Also data type like atomic trades or klines.

Parameters:

• filename – A filename with expected format. Example: LTCBTC 1m Europe-Madrid optional comment klines 1691332020000 1691391959999.csv

• expected_data_type (str) – Expected data type. Example: “klines” or “aggTrades” or “atomicTrades”

• expected_interval (str) – Expected tick interval. Example: “1m” or “1h”.

• expected_symbol (str) – Expected symbol. Example: “LTCBTC”

• expected_timezone (str) – Expected timezone. Example: “Europe/Madrid”

Returns:

A dict with metadata.

read_csv_to_dataframe(filename: str, col_sep: str = ',', index_col: str = None, index_time_zone: str = None, symbol: str = None, secondary_index_col: str = None) → DataFrame

Creates a csv file from a dataframe.

Parameters:

• filename (str) – The file name with or without path.

• col_sep (str) – Column separator. Default is “,”

• index_col (bool) – Name of the column to use as index. Default is None.

• secondary_index_col (str) – In case of duplicated index, it will be used as secondary criterio for sorting data.

• index_time_zone (str) – Time zone to use in index. Default is None.

• symbol (str) – Symbol to use in index name. Default is None.

Return pd.DataFrame:

A dataframe with data in columns using file rows header.

read_file(filename: str) → list

Read a file to a list of strings each line.

Return list:

list with a string each row in the file.

save_file(filename: str, data: list, mode='w') → None

Save a new file from a list of lists each line.

Parameters:

• filename (str) – a file name to save.

• data (list) – Data in a list of strings each line.

• mode (str) – ‘w’ to rewrite full file or ‘a’ to append to existing file.

select_file(path='.', extension='csv', symbol: str = None, tick_interval: str = None, name_filter: str = None) → str

Selects from files in the path with the extension passed.

Parameters:

• path (str) – Path to search files.

• extension (str) – Extension of interesting files to select.

• symbol (str) – Symbol to filter files just to alert if file has other symbol.

• tick_interval (str) – Tick interval to filter files just to alert if file has other tick interval.

• name_filter (str) – Filter to apply to files. Example: “klines” or “aggTrades” or “atomicTrades”.

Return str:

a filename.

get_database_password() → str

Get the PostgreSQL password in plain text from panzer (encrypted on disk).

Return str:

The PostgreSQL password.

add_any_key(key: str, key_name: str) → None

Store any credential in panzer’s CredentialManager (~/.panzer_creds).

Sensitive names (containing secret, api_key, password or _id) are encrypted automatically; the rest are stored as plain text.

Example to add a database password:

from binpan.storage.files import add_any_key

add_any_key(key="xxxxxxxxxx", key_name="postgresql_password")

Parameters:

• key (str) – Any API key, token or password value.

• key_name (str) – Name to store and later retrieve it by.