Data ingestion

Data ingestion#

Acoustics#

echopop.ingest.sv.ingest_echoview_sv(sv_path: Path, center_frequencies: dict[str, float] | None = None, transect_pattern: str | None = None, aggregate_method: Literal['cells', 'interval', 'transect'] = 'cells', impute_coordinates: bool = True) → tuple[DataFrame, DataFrame]#

Complete ingestion pipeline for Echoview Sv export data.

This is the main entry point for processing Echoview volume backscattering strength exports. It handles file discovery, data loading, coordinate imputation, frequency filtering, and spatial aggregation to produce analysis-ready acoustic datasets.

Parameters:
sv_pathpathlib.Path

Directory path containing Echoview CSV export files

center_frequenciesDict[str, float], optional

Dictionary mapping target frequencies (Hz) to threshold dictionaries with ‘min’ and ‘max’ Sv values in dB. If None, uses all frequencies with permissive thresholds.

transect_patternstr, optional

Regular expression pattern to extract transect numbers from filenames. Should contain a capture group for the transect number

aggregate_methodLiteral[“cells”, “interval”, “transect”], default=”cells”

Spatial aggregation method for acoustic data integration

impute_coordinatesbool, default=True

Whether to interpolate missing latitude/longitude coordinates

Returns:
tuple[pandas.DataFrame, pandas.DataFrame or None]

  • sv_integrated: Spatially aggregated acoustic data with pandas.MultiIndex columns organized by measurement type and frequency.

  • sv_coordinates: Coordinate reference data for spatial analysis, or None if coordinates unavailable.

Raises:
FileNotFoundError

If sv_path does not exist or contains no files

Notes

Complete processing workflow:

  1. File Discovery: Recursively find all CSV files in directory

  2. Transect Mapping: Extract transect numbers using regex pattern

  3. Data Loading: Read and concatenate all Sv export files

  4. Quality Control: Sort data and validate structure

  5. Frequency Filtering: Select target frequencies if specified

  6. Integration: Apply spatial aggregation with thresholding

  7. Output Formatting: Structure data for downstream analysis

The function automatically converts frequency units from Hz to kHz to match Echoview export conventions.

Examples

>>> sv_path = Path("/data/acoustic_exports/")
>>> frequencies = {18000: {"min": -90, "max": -50},
...                38000: {"min": -85, "max": -45}}
>>> sv_data, coords = ingest_echoview_sv(
...     sv_path, frequencies, r"transect_(\d+)", "interval"
... )
echopop.ingest.nasc.merge_echoview_nasc(file_directory: Path, filename_transect_pattern: str = 'T(\\d+)', default_transect_spacing: float = 10.0, default_latitude_threshold: float = 60.0, impute_coordinates: bool = True) → tuple[DataFrame, DataFrame]#

Ingest and merge all Echoview NASC files (intervals, cells, layers).

Parameters:
file_directoryPath

Directory containing Echoview export files (\\*.csv).

filename_transect_pattern: str, default = ``r”T(\d+)”``

Regular expression used for extracting the transect number from the filename.

default_transect_spacingfloat, default = 10.

Default spacing (nmi) to impute where missing.

default_latitude_thresholdfloat, default = 60.

Default latitude threshold used for determining how far north transect spacings should be calculated versus using the default value.

impute_coordinatesbool

Instruct whether bad spatial coordinates should be imputed or not

Returns:
Tuple[pandas.DataFrame, pandas.DataFrame]

A tuple containing two pandas.DataFrames:

  • df_intervals: The processed intervals DataFrame with added transect spacing

  • merged_exports_df: The merged DataFrame from intervals, cells, and layers

echopop.ingest.nasc.consolidate_echvoiew_nasc(nasc_data: DataFrame, interval_data: DataFrame, region_class_names: list[str], impute_region_ids: bool = True, transect_region_haul_key: DataFrame | None = None, haul_uid_config: dict[str, Any] | None = None) → DataFrame#

Consolidate Echoview NASC data with interval and region information.

Parameters:
nasc_datapandas.DataFrame

DataFrame containing merged Echoview data with columns:

  • 'region_class' : Region classification names

  • 'region_id' : Region identifier

  • 'nasc' : Nautical area scattering coefficient

  • 'transect_num' : Transect number

  • 'interval' : Interval identifier

interval_datapandas.DataFrame

DataFrame containing interval information with columns:

  • 'interval' : Interval identifier

  • 'transect_num' : Transect number

  • 'distance_s' : Starting distance

  • 'distance_e' : Ending distance

  • 'latitude' : Latitude coordinates

  • 'longitude' : Longitude coordinates

  • 'transect_spacing' : Spacing between transects

region_class_namesList[str]

List of region class names to include in the analysis

impute_region_idsbool, optional

Whether to impute region IDs for overlapping regions, by default True

transect_region_haul_keypandas.DataFrame, optional

DataFrame containing haul information to merge, by default None

haul_uid_configDict[str, Any]

Optional keyword arguments to override defaults or DataFrame values:

  • ship_id (dict): Region-specific IDs, e.g., {‘US’: 10, ‘CAN’: 20}.

  • survey_id (dict): Region-specific IDs, e.g., {‘US’: 1, ‘CAN’: 2}.

  • species_id (int/str): A global species code override.

  • haul_offset (int/float): A value subtracted from ‘haul_num’ for records identified as ‘CAN’ (where haul_num - offset >= 0).

Returns:
pandas.DataFrame

Consolidated DataFrame containing:

  • 'transect_num' : Transect number

  • 'region_id' : Region identifier (999 for NaN)

  • 'distance_s' : Starting distance

  • 'distance_e' : Ending distance

  • 'latitude' : Latitude coordinates

  • 'longitude' : Longitude coordinates

  • 'transect_spacing' : Spacing between transects

  • 'layer_mean_depth' : Mean layer depth

  • 'layer_height' : Layer height

  • 'bottom_depth' : Bottom depth

  • 'nasc' : Summed nautical area scattering coefficient

  • 'haul_num' : Haul number (0 for NaN)

  • 'uid' : Unique ship-survey-species-haul identifier

Notes

All numeric columns ('nasc', 'layer_mean_depth', 'layer_height', 'bottom_depth') are filled with 0.0 for NaN values.

echopop.ingest.nasc.generate_transect_region_haul_key(region_data: DataFrame, filter_list: list[str]) → DataFrame#

Filter DataFrame by region class patterns and create a mapping.

Filters the DataFrame to include only rows with region classes in the provided filter list and creates a mapping of unique regions.

Parameters:
region_datapandas.DataFrame

pandas.DataFrame with processed region data

filter_listList[str]

List of region class names to include in the filter

Returns:
pandas.DataFrame

Filtered and grouped pandas.DataFrame containing unique region mappings

Notes

The returned pandas.DataFrame is grouped by 'transect_num', 'haul_num', and 'region_id', with first instances of 'region_class' and 'region_name', sorted by 'haul_num'.

echopop.ingest.nasc.process_region_names(nasc_cells: DataFrame, region_name_expr: dict, can_haul_offset: int | None = None) → DataFrame#

Process region names in a DataFrame using regex patterns.

Coordinates the extraction and processing of region name components from a pandas.DataFrame according to specified patterns, with optional filtering and mapping.

Parameters:
nasc_cellspandas.DataFrame

A pandas.DataFrame comprising NASC export data that must include the column 'region_name' to function.

region_name_exprDict

Dictionary of pattern specifications for component extraction:

  • Keys are component names (e.g., 'REGION_CLASS', 'HAUL_NUM', 'COUNTRY')

  • Values are either:

    • Dict mapping labels to regex pattern strings

    • Set of regex pattern strings

can_haul_offsetOptional[int], Default None

Offset to add to haul numbers for Canadian regions

Returns:
pandas.DataFrame

If filter_list is provided: a mapping of unique regions filtered by region class. Otherwise, the original pandas.DataFrame with extracted components added.

echopop.ingest.nasc.read_nasc_file(filename: str | Path, sheetname: str, impute_coordinates: bool = True, column_name_map: dict[str, str] | None = None, haul_uid_config: dict[str, Any] | None = None)#

Read NASC data from a consolidated XLSX file.

Parameters:
filenamestr or pathlib.Path

Path to the Excel file

sheetnamestr

Name of the sheet to read

impute_coordinatesbool

Instruct whether bad spatial coordinates should be imputed or not

column_name_mapdict, optional

Dictionary mapping original column names to new column names

haul_uid_configDict[str, Any]

Optional keyword arguments to override defaults or DataFrame values:

  • ship_id (dict): Region-specific IDs, e.g., {‘US’: 10, ‘CAN’: 20}.

  • survey_id (dict): Region-specific IDs, e.g., {‘US’: 1, ‘CAN’: 2}.

  • species_id (int/str): A global species code override.

  • haul_offset (int/float): A value subtracted from ‘haul_num’ for records identified as ‘CAN’ (where haul_num - offset >= 0).

Returns:
pandas.DataFrame

Cleaned DataFrame with renamed columns and imputed coordinates

Examples

>>> column_map = {"transect": "transect_num", "region id": "region_id"}
>>> df = read_nasc_file("data.xlsx", "Sheet1", column_map)

>>> # Without column mapping
>>> df = read_nasc_file("data.xlsx", "Sheet1")
echopop.ingest.nasc.read_afsc_nasc_file(filename: str | Path, sheetname: str, impute_coordinates: bool = True, column_name_map: dict[str, str] | None = None)#

Read AFSC NASC data from a consolidated XLSX file.

This is for the AFSC files that are missing the haul number column.

Parameters:
filenamestr or pathlib.Path

Path to the Excel file

sheetnamestr

Name of the sheet to read

impute_coordinatesbool

Instruct whether bad spatial coordinates should be imputed or not

column_name_mapdict, optional

Dictionary mapping original column names to new column names

Returns:
pandas.DataFrame

Cleaned DataFrame with renamed columns and imputed coordinates

Examples

>>> column_map = {"transect": "transect_num", "region id": "region_id"}
>>> df = read_nasc_file("data.xlsx", "Sheet1", column_map)

>>> # Without column mapping
>>> df = read_nasc_file("data.xlsx", "Sheet1")
echopop.ingest.nasc.read_transect_region_haul_key(filename: Path, sheetname: str, column_name_map: dict[str, str] | None = None) → DataFrame#

Load the key that maps hauls to export regions to transect numbers.

This function reads a CSV or Excel file containing the mapping between transect numbers, region IDs, and haul numbers. It can handle both file formats and allows column renaming.

Parameters:
filenamePath

Path to the CSV or Excel file containing the mapping data.

sheetnamestr

Name of the sheet to read (only used for Excel files).

column_name_mapOptional[Dict[str, str]], default None

Dictionary for renaming columns, where keys are original column names and values are new column names.

Returns:
pandas.DataFrame

DataFrame containing only the columns "transect_num", "region_id", and "haul_num".

Notes

The input file must contain columns that can be mapped to "transect_num", "region_id", and "haul_num", either directly or via the column_name_map.

Biological#

echopop.ingest.load_biodata_views(biodata_filepaths: dict[str, Path], column_name_map: dict[str, str] = None, survey_subset: dict | None = None, biodata_label_map: dict[str, dict] | None = None, haul_uid_config: dict[str, Any] | None = None) → dict[str, DataFrame]#

Load biological data from a database view.

Load biological data from materialized views of catch data and a file including specimen-only values.

Parameters:
biodata_filepathspathlib.Path

A dictionary of filepaths to *.csv files containing biological data

column_name_mapdict, optional

Dictionary mapping original column names to new column names (e.g., {"frequency": "length_count", "haul": "haul_num"})

survey_subsetdict, optional

Subset dictionary containing ships and species_code for filtering Format: {"ships": {ship_id: {"survey": survey_id, "haul_offset": offset}}, "species_code": [codes]}

biodata_label_mapdict, optional

Dictionary mapping column names to value replacement dictionaries (e.g., {"sex": {1: "male", 2: "female", 3: "unsexed"}})

haul_uid_configDict[str, Any]

Optional keyword arguments to override defaults or DataFrame values:

  • ship_id (dict): Region-specific IDs, e.g., {'US': 10, 'CAN': 20}.

  • survey_id (dict): Region-specific IDs, e.g., {'US': 1, 'CAN': 2}.

  • species_id (int/str): A global species code override.

  • haul_offset (int/float): A value subtracted from 'haul_num' for records identified as ‘CAN’ (where haul_num - offset >= 0).

Returns:
dict

Dictionary containing processed biological DataFrames keyed by dataset name

echopop.ingest.load_biological_data(biodata_filepath: Path, biodata_sheet_map: dict[str, str], column_name_map: dict[str, str] = None, survey_subset: dict | None = None, biodata_label_map: dict[str, dict] | None = None, haul_uid_config: dict[str, Any] | None = None) → dict[str, DataFrame]#

Load biological data from a single Excel file with multiple sheets.

Parameters:
biodata_filepathpathlib.Path

Path to the Excel file containing biological data

biodata_sheet_mapdict

Dictionary mapping dataset names to sheet names (e.g., {"specimen": "biodata_specimen", "length": "biodata_length", "catch": "biodata_catch"})

column_name_mapdict, optional

Dictionary mapping original column names to new column names (e.g., {"frequency": "length_count", "haul": "haul_num"})

survey_subsetdict, optional

Subset dictionary containing ships and species_code for filtering Format: {"ships": {ship_id: {"survey": survey_id, "haul_offset": offset}}, "species_code": [codes]}

biodata_label_mapdict, optional

Dictionary mapping column names to value replacement dictionaries (e.g., {"sex": {1: "male", 2: "female", 3: "unsexed"}})

haul_uid_configDict[str, Any]

Optional keyword arguments to override defaults or DataFrame values:

  • ship_id (dict): Region-specific IDs, e.g., {‘US’: 10, ‘CAN’: 20}.

  • survey_id (dict): Region-specific IDs, e.g., {‘US’: 1, ‘CAN’: 2}.

  • species_id (int/str): A global species code override.

  • haul_offset (int/float): A value subtracted from ‘haul_num’ for records identified as ‘CAN’ (where haul_num - offset >= 0).

Returns:
dict

Dictionary containing processed biological DataFrames keyed by dataset name

Examples

>>> sheet_map = {"catch": "biodata_catch", "length": "biodata_length"}
>>> subset = {"ships": {160: {"survey": 201906}}, "species_code": [22500]}
>>> col_map = {"frequency": "length_count", "haul": "haul_num"}
>>> label_map = {"sex": {1: "male", 2: "female", 3: "unsexed"}}
>>> bio_data = load_biological_data("biodata.xlsx", sheet_map, col_map, subset, label_map)

Strata and Geostrata#

echopop.ingest.load_geostrata(geostrata_filepath: Path, geostrata_sheet_map: dict[str, str], column_name_map: dict[str, str] = None) → dict[str, DataFrame]#

Load geographic stratification data from an Excel file with multiple sheets.

Parameters:
geostrata_filepathpathlib.Path

Path to the Excel file containing geographic stratification data

geostrata_sheet_mapdict

Dictionary mapping stratification types to sheet names (e.g., {"inpfc": "INPFC", "ks": "stratification1"})

column_name_mapdict, optional

Dictionary mapping original column names to new column names (e.g., {"Latitude (upper limit)": "northlimit_latitude", "stratum": "stratum_num"})

Returns:
dict

Dictionary containing geographic stratification pandas.DataFrames keyed by stratification type, each with consolidated latitude intervals from INPFC and KS strata assignments

echopop.ingest.join_geostrata_by_latitude(data: DataFrame | dict[str, DataFrame], geostrata: DataFrame, stratum_name: str = 'stratum_num') → DataFrame | dict[str, DataFrame]#

Join geographic stratification data by latitude intervals.

Parameters:
datapandas.DataFrame or Dict[str, pandas.DataFrame]

DataFrame or dictionary of DataFrames with latitude information

geostratapandas.DataFrame

Geographic stratification DataFrame with latitude boundaries and stratum info

stratum_namestr, default=”stratum_num”

Name of the column containing stratum information

Returns:
pandas.DataFrame or Dict[str, pandas.DataFrame]

Same type as input data with stratification added

echopop.ingest.load_strata(strata_filepath: Path, strata_sheet_map: dict[str, str], column_name_map: dict[str, str] = None, haul_uid_config: dict[str, Any] | None = None) → dict[str, DataFrame]#

Load stratification data from an Excel file with multiple sheets.

Parameters:
strata_filepathPath

Path to the Excel file containing stratification data

strata_sheet_mapdict

Dictionary mapping stratification types to sheet names (e.g., {"inpfc": "INPFC", "ks": "stratification1"})

column_name_mapdict, optional

Dictionary mapping original column names to new column names (e.g., {"fraction_hake": "nasc_proportion", "haul": "haul_num"})

haul_uid_configDict[str, Any]

Optional keyword arguments to override defaults or DataFrame values:

  • ship_id (dict): Region-specific IDs, e.g., {‘US’: 10, ‘CAN’: 20}.

  • survey_id (dict): Region-specific IDs, e.g., {‘US’: 1, ‘CAN’: 2}.

  • species_id (int/str): A global species code override.

  • haul_offset (int/float): A value subtracted from ‘haul_num’ for records identified as ‘CAN’ (where haul_num - offset >= 0).

Returns:
dict

Dictionary containing stratification pandas.DataFrames keyed by stratification type

Examples

>>> sheet_map = {"inpfc": "INPFC", "ks": "stratification1"}
>>> col_map = {"fraction_hake": "nasc_proportion", "haul": "haul_num"}
>>> strata_data = load_stratification("strata_file.xlsx", sheet_map, col_map)
echopop.ingest.join_strata_by_haul(data: DataFrame | dict[str, DataFrame], strata: dict[str, DataFrame], default_stratum: float = 0.0, stratum_name: str = 'stratum_num') → DataFrame | dict[str, DataFrame]#

Join stratification data by haul number or other matching column.

Parameters:
datapandas.DataFrame or Dict[str, pandas.DataFrame]

DataFrame or dictionary of DataFrames to join with strata

stratapandas.DataFrame

Specific stratification DataFrame with stratum-haul key information

default_stratumfloat

Default stratum value when there are no matching/corresponding values

stratum_namestr, default=”stratum_num”

Name of the column containing stratum information

Returns:
pandas.DataFrame or Dict[str, pandas.DataFrame]

Same type as input data with stratification added

echopop.ingest.join_strata_by_uid(data: DataFrame | dict[str, DataFrame], strata: dict[str, DataFrame], default_stratum: float = 0.0, stratum_name: str = 'stratum_num') → DataFrame | dict[str, DataFrame]#

Join data and stratification definitions using an unique identifier.

Parameters:
datapandas.DataFrame or Dict[str, pandas.DataFrame]

DataFrame or dictionary of DataFrames to join with strata

stratapandas.DataFrame

Specific stratification DataFrame with stratum-haul key information

default_stratumfloat

Default stratum value when there are no matching/corresponding values

stratum_namestr, default=”stratum_num”

Name of the column containing stratum information

Returns:
pandas.DataFrame or Dict[str, pandas.DataFrame]

Same type as input data with stratification added

Spatial#

echopop.ingest.load_isobath_data(isobath_filepath: str | Path, sheet_name: str, column_name_map: dict[str, str] | None = None) → DataFrame#

Load isobath data from an Excel file.

Parameters:
isobath_filepathstr or pathlib.Path

Path to the Excel file containing mesh data

sheet_namestr

Name of the sheet containing the mesh data

column_name_mapdict, optional

Dictionary mapping original column names to new column names

Returns:
pandas.DataFrame

DataFrame containing the isobath data with longitude and latitude

Examples

>>> isobath_df = load_isobath_data("isobath_file.xlsx", "Sheet1")
>>> isobath_df = load_isobath_data("isobath_file.xlsx", column_name_map={"latitude_200m":
"latitude"})
echopop.ingest.load_mesh_data(mesh_filepath: str | Path, sheet_name: str, column_name_map: dict[str, str] | None = None) → DataFrame#

Load mesh data from an Excel file.

Parameters:
mesh_filepathstr or pathlib.Path

Path to the Excel file containing mesh data

sheet_namestr

Name of the sheet containing the mesh data

column_name_mapdict, optional

Dictionary mapping original column names to new column names

Returns:
pandas.DataFrame

DataFrame containing the mesh data with centroid coordinates and fractions

Examples

>>> mesh_df = load_mesh_data("mesh_file.xlsx", "Grid")
>>> mesh_df = load_mesh_data("mesh_file.xlsx", column_name_map={"centroid_latitude":
"latitude"})
echopop.ingest.load_kriging_variogram_params(geostatistic_params_filepath: Path, sheet_name: str, column_name_map: dict[str, str] | None = None) → tuple[dict, dict]#

Load kriging and variogram parameters from Excel file.

Parameters:
geostatistic_params_filepathpathlib.Path

Path to the Excel file containing kriging parameters

sheet_namestr

Name of the sheet to load

column_name_mapdict, optional

Dictionary mapping original column names to new column names

Returns:
tuple

Tuple containing kriging and variogram parameter dictionaries