Survey

Survey#

Biological#

echopop.survey.biology.compute_abundance(transect_data: DataFrame, exclude_filter: dict[str, str] = None, number_proportions: dict[str, Dataset] | None = None) → None#

Convert number density estimates in abundances.

Parameters:
transect_datapd.DataFrame

DataFrame containing transect data with number densities already computed. Must include:

  • ‘number_density’: Number of animals per unit area (animals/nmi²)

  • ‘area_interval’: Area of each sampling interval (nmi²)

stratify_byList[str], default []

Column name to use for stratification when aligning with the number proportions dictionary, if specified.

group_byList[str], default []

Grouping columns to apply to number density and abundances (e.g. sex). This will produce additional columns formatted as number_density_{group} and abundance_{group}.

exclude_filterDict[str, str], default {}

Dictionary specifying which population segments to exclude and redistribute. Keys are column/index names, values are the categories to exclude.

number_proportionsOptional[Dict[str, pd.DataFrame]], default None

When provided, the number densities and abundances will be reapportioned according to the defined number proportions. If group_by is not empty, then the resulting DataFrame will include additional columns for each group as well as the overall number density and abundance.

Returns:
None

Function modifies the transect DataFrame in-place.

Notes

When number_proportions is provided, the function first calculates base abundance as area_interval * number_density, then applies proportions to create grouped abundance columns. The original ‘abundance’ column represents the total.

echopop.survey.biology.compute_biomass(transect_data: DataFrame, stratum_weights: DataArray | float | None = None) → None#

Convert number density and abundance estimates to biomass density and total biomass.

Convert number density and abundance estimates to biomass density and total biomass, respectively. This is done by aligning transect data with either a scalar average weight that is applied globally, or group-specific averaged weights (e.g., by stratum and sex). When grouped, the biomass density and biomass estimates are computed across all defined groups.

Parameters:
transect_datapd.DataFrame

DataFrame containing transect data with number densities already computed. Must include:

  • ‘number_density’: Number of animals per unit area (animals/nmi²)

  • ‘abundance’: Total number of animals in each interval (animals)

stratum_weightsxr.DataArray, float, optional

A float or DataArray containing average weight estimates. When this is a float, the quantity is applied to all values uniformly. When it is a DataArray, then the coordinates are used to partition biomass estimates into separate groups via: biomass_density_{coordinate} (e.g., ‘biomass_density_male’) and biomass_{coordinate} (e.g., ‘biomass_female’).

Returns:
None

Function modifies the transect DataFrame in-place.

echopop.survey.biology.fit_length_weight_regression(data: DataFrame) → Series#

Fit a log-linear length-weight regression to biological data.

This function fits a linear regression to log10-transformed length and weight data, following the standard allometric relationship: log(weight) = a + b*log(length). The function can be used standalone or as part of a groupby operation.

Parameters:
datapd.DataFrame

DataFrame containing ‘length’ and ‘weight’ columns.

Returns:
pd.Series

Series with regression coefficients: - ‘slope’: The slope coefficient (b) from log(weight) = a + b*log(length) - ‘intercept’: The intercept coefficient (a) from log(weight) = a + b*log(length)

Examples

>>> # Standalone usage
>>> coeffs = fit_length_weight_regression(length_weight_dataset)

>>> # With groupby
>>> grouped_coeffs = data.groupby('sex').apply(fit_length_weight_regression)
echopop.survey.biology.length_binned_weights(data: DataFrame, length_bins: ndarray, regression_coefficients: Series | DataFrame, impute_bins: bool = True, minimum_count_threshold: int = 0) → DataArray#

Compute length-binned average weights using regression coefficients and observed data.

This function calculates fitted weights for length bins by combining modeled weights (from length-weight regression) with observed mean weights. For bins with sufficient sample sizes, observed means are used; for bins with low sample sizes, modeled weights are used if imputation is enabled.

Parameters:
datapd.DataFrame

Specimen data containing ‘length’, ‘weight’, and ‘length_bin’ columns. The ‘length_bin’ column must already exist in the data.

length_distributionpd.DataFrame

DataFrame with length bin information, containing ‘bin’ and ‘interval’ columns.

regression_coefficientspd.Series or pd.DataFrame

Length-weight regression coefficients from fit_length_weight_regression(). If DataFrame, represents grouped coefficients (e.g., by sex).

impute_binsbool, default True

Whether to use modeled weights for bins with insufficient data. If False, only observed means are used regardless of sample size.

minimum_count_thresholdint, default 0

Minimum number of specimens required to use observed mean instead of modeled weight. Only relevant when impute_bins=True.

Returns:
xr.DataArray

Fitted weights indexed by length bin and grouping dimensions. The returned DataArray has:

  • Dimensions:

    • ‘length_bin’: Length-bin intervals.

    • One dimension per grouping variable implied by regression_coefficients (e.g., sex). Present only when grouped coefficients are supplied.

  • Data:

    • ‘weight_fitted’: Fitted weight values for each coordinate combination.

Examples

>>> # Single coefficient set
>>> coeffs = fit_length_weight_regression(specimen_data)
>>> fitted = compute_binned_weights(specimen_data, length_dist, coeffs)

>>> # Grouped coefficients (e.g., by sex)
>>> sex_coeffs = specimen_data.groupby('sex').apply(fit_length_weight_regression)
>>> fitted = compute_binned_weights(specimen_data, length_dist, sex_coeffs,
...                                minimum_count_threshold=5)

>>> # No imputation - use only observed means
>>> fitted = compute_binned_weights(specimen_data, length_dist, coeffs,
...                                impute_bins=False)
echopop.survey.biology.quantize_length_data(df, group_columns: list[str])#

Process DataFrame to ensure it has ‘length’ and ‘length_count’ columns.

Aggregates fish length data by grouping variables and length, either counting occurrences (if no length_count exists) or summing existing counts.

Parameters:
dfpd.DataFrame

DataFrame containing length data. Must have a ‘length’ column. Optionally can have ‘length_count’ column with existing counts.

group_columnsList[str]

List of column names to group by before aggregating length counts. Common examples: [‘stratum’, ‘haul_num’, ‘sex’]

Returns:
pd.DataFrame

DataFrame with grouping columns, ‘length’, and ‘length_count’ columns. Index will be a MultiIndex of group_columns + [‘length’].

Notes

This function automatically detects whether to count fish (size operation) or sum existing counts based on the presence of a ‘length_count’ column.

The resulting DataFrame will have a MultiIndex with group_columns + [‘length’] and a single ‘length_count’ column containing the aggregated counts.

Examples

>>> # Data without existing counts - will count occurrences
>>> specimen_df = pd.DataFrame({
...     'stratum': [1, 1, 1, 2, 2],
...     'sex': ['M', 'M', 'F', 'F', 'M'],
...     'length': [20.5, 20.5, 22.1, 18.3, 20.5]
... })
>>> result = quantize_length_data(specimen_df, ['stratum', 'sex'])
>>> print(result)
                       length_count
stratum sex length
1       F   22.1               1
        M   20.5               2
2       F   18.3               1
        M   20.5               1

>>> # Data with existing counts - will sum them
>>> length_df = pd.DataFrame({
...     'stratum': [1, 1, 2],
...     'length': [20.5, 22.1, 20.5],
...     'length_count': [5, 3, 2]
... })
>>> result = quantize_length_data(length_df, ['stratum'])
>>> print(result)
               length_count
stratum length
1       20.5              5
        22.1              3
2       20.5              2

Distributions and proportions#

echopop.survey.proportions.compute_binned_counts(data: DataFrame, groupby_cols: list[str], count_col: str, agg_func: str = 'size') → DataArray#

Compute binned counts with grouping.

Parameters:
datapd.DataFrame

Input DataFrame

groupby_colslist

Columns to group by

count_colstr

Column to aggregate

agg_funcstr, default “size”

Aggregation function to apply: “size”, “sum”, “count”, etc. Column-value pairs to exclude. Format: {column: value_to_exclude}

Returns:
xr.DataArray

Grouped counts with coordinates based on the column names provided in ‘groupby_cols’.

echopop.survey.proportions.number_proportions(data: DataArray | Dataset, stratum_dim: str | None = None, exclude_filters: dict | None = None) → DataArray | Dataset#

Calculate number proportions from one or more xarray DataArrays or Datasets.

This function computes proportions across all provided xarray objects. It can handle different xarray structures, as long as they all have a 'count' variable and share the grouping columns.

Parameters:
dataUnion[xr.DataArray, xr.Dataset]

Either a single xarray.DataArray or an xarray.Dataset containing multiple named xarray.DataArray objects.

stratum_dimstr, optional

Stratification dimension name to group by for calculating totals (e.g., "stratum_ks").

exclude_filtersdict, default {}

Filters to exclude coordinates from xarray objects. For a single DataArray, this should be a dict like {“sex”: “unsexed”}; for a Dataset, a dict of dicts keyed by variable name.

Returns:
Union[xr.DataArray, Dict[xr.Dataset]]

If only one DataArray is provided, returns that xarray.DataArray with added proportion coordinates. If a Dataset is provided, returns a dictionary of xarray.Datasets for each variable within data.

Notes

The function calculates a single proportion variable:

  • “proportion”: Overall proportion (count divided by the sum of counts across all DataArrays within the original Dataset)

Examples

>>> # Single DataArray
>>> result = number_proportions(aged_counts)
>>>
>>> # Dataset
>>> dataset_input = Dataset({"aged": aged_counts, "unaged": unaged_counts})
>>> result = number_proportions(dataset_input)
echopop.survey.proportions.binned_weights(length_data: DataFrame, interpolate_regression: bool = True, group_columns: list | None = None, length_weight_data: DataArray | None = None, include_filter: dict[str, Any] | None = None) → DataArray#

Process length-weight data and return an xarray.DataArray of weights binned by group_columns.

This function creates a DataArray of weights by length bins and optionally other grouping variables. If interpolate_regression is True, weights are interpolated using a regression model provided as a DataArray. If False, weights are taken directly from the length_dataset. Filtering is applied using include_filter, which keeps only rows matching the filter.

Parameters:
length_datasetpd.DataFrame

Dataset with length measurements. Must contain ‘length_bin’ and ‘length_count’ columns when using interpolation, or direct ‘weight’ values when not interpolating.

interpolate_regressionbool, default True

Whether to use interpolation for weights. If True, interpolates weights based on length-weight relationships from length_weight_dataset. If False, uses existing weight values in length_dataset.

group_columnslist, default [“length_bin”]

Columns to use as coordinates in the output DataArray.

length_weight_datasetxr.DataArray, optional

DataArray with length-weight relationships. Required when interpolate_regression=True. Must contain ‘length_bin’ as a coordinate.

include_filterDict[str, Any], optional

Filter to apply to both datasets (e.g., to include only certain sexes). Rows are kept if they match the filter.

Returns:
xr.DataArray

DataArray of weights binned by group_columns.

Raises:
ValueError

If interpolate_regression=True but length_weight_dataset is None or missing ‘length_bin’.

Notes

  • The output DataArray will have coordinates as specified by group_columns.

  • Filtering is performed before binning/interpolation.

  • If interpolate_regression is True, interpolation is performed using the regression model in length_weight_dataset, which is first converted to a DataFrame for compatibility.

  • If interpolate_regression is False, weights are taken directly from length_dataset.

Examples

>>> # Using interpolation with regression model
>>> weights = binned_weights(
...     length_dataset=length_freq_df,
...     length_weight_dataset=length_weight_da,
...     interpolate_regression=True,
...     group_columns=["length_bin", "sex"],
...     include_filter={"sex": ["female", "male"]}
... )
>>> # Using direct weights (no interpolation)
>>> weights = binned_weights(
...     length_dataset=specimen_df,
...     interpolate_regression=False,
...     group_columns=["length_bin", "sex"],
...     include_filter={"sex": ["female", "male"]}
... )
echopop.survey.proportions.stratum_averaged_weight(number_proportions: dict[str, Dataset] | Dataset, length_weight_data: DataArray, stratum_dim: str | None = None) → DataArray#

Calculate stratum- and group-specific average weights using xarray inputs.

This function combines length-at-age or length-only distributions from multiple groups (e.g., aged and unaged fish) to produce weighted average weights by group and stratum, using xarray objects for all calculations. It accounts for the proportions of each group and correctly weights their contributions to the final average.

Parameters:
number_proportionsDict[str, xr.Dataset]

Dictionary mapping group names to xarray Datasets containing proportion data. Each Dataset must have at least one variable representing proportions and coordinates for all relevant grouping and binning dimensions.

length_weight_dataxr.DataArray

xarray DataArray containing fitted weights, with coordinates matching the grouping and binning dimensions used in the proportion Datasets.

stratum_dimstr, optional

Stratification dimension name to use for grouping (e.g., "stratum_ks"). If None, no stratification grouping is applied.

Returns:
xr.DataArray

DataArray with average weights for each stratum and group, indexed by stratum_dim and groupings.

Notes

  • The function assumes that the length bins in number_proportions and length_weight_data match.

  • The function requires an ‘all’ group category in length_weight_data for calculating combined weights.

  • Missing strata or group categories will be excluded from the final results.

  • All calculations are performed using xarray and pandas interconversion for flexibility.

Examples

>>> # Prepare xarray Datasets for proportions (see proportions.number_proportions)
>>> number_proportions = {'aged': ds_aged, 'unaged': ds_unaged}
>>> # Prepare xarray DataArray for weights (see biology.length_binned_weights)
>>> length_weight_data = da_binned_weights
>>> # Calculate average weights
>>> weights = stratum_averaged_weight(number_proportions, length_weight_data,
stratum_dim='stratum_num')
>>> print(weights)
<xarray.DataArray ...>
echopop.survey.proportions.weight_proportions(weight_data: DataArray, catch_data: DataFrame | dict[str, DataFrame], stratum_dim: str | None = None, proportion_reference: Literal['catch', 'catch_plus_specimen'] = 'catch') → DataArray#

Calculate stratified weight proportions using xarray and pandas inputs.

This function computes the proportional weight distribution for each group relative to the total weights across all strata or groupings, using an xarray DataArray for biological weights and a pandas DataFrame for catch weights. All grouping is handled dynamically based on the provided stratum_dim.

Parameters:
weight_dataxr.DataArray

xarray DataArray containing weight distributions for all groups, with dimensions including stratum_dim.

catch_datapd.DataFrame

DataFrame with catch data, including columns for stratum_dim and “weight”.

stratum_dimstr, optional

Stratification dimension name to group by (e.g., "stratum_ks").

proportion_referenceLiteral[“catch”, “catch_plus_specimen”], default ‘catch’

Determines the denominator for the proportions calculation:

  • “catch”: proportions are relative to the summed catch weights only (default).

  • “catch_plus_specimen”: proportions are relative to the sum of catch weights and individual fish weights

Returns:
xr.DataArray

DataArray containing the calculated weight proportions, indexed by all grouping dimensions present in the input data.

Notes

  • No column or dimension names are hard-coded; all logic is dynamic.

  • The function assumes that grouping columns are consistent between weight_data and catch_data.

  • Missing or extra group/category combinations are handled automatically by xarray/pandas.

Examples

>>> weight_props = weight_proportions(
...     weight_data=ds_da_weight_dist["aged"],
...     catch_data=dict_df_bio["catch"],
...     stratum_dim="stratum_ks",
...     proportion_reference="catch"
... )
>>> print(weight_props)
<xarray.DataArray ...>
echopop.survey.proportions.fitted_weight_proportions(weight_data: DataArray, aged_weight_proportions: Dataset, number_proportions: Dataset, binned_weights: DataArray, stratum_dim: str | None = None) → Dataset#

Calculate weight proportions based on fitted weights with reference-value adjustments.

This function computes comprehensive weight proportions for a group, accounting for reference proportions, length-binned proportions, and fitted weight tables. All grouping and dimension logic is handled dynamically based on the provided xarray objects and stratum_dim.

Parameters:
weight_dataxr.DataArray

xarray DataArray of scaled weights for the group, with dimensions including stratum_dim.

aged_weight_proportionsxr.Dataset

xarray Dataset of reference aged weight proportions for comparison, with compatible dimensions.

number_proportionsxr.Dataset

xarray Dataset of number proportions by relevant grouping factors.

binned_weightsxr.DataArray

xarray DataArray of fitted weights by length bins (and possibly other groupings).

stratum_dimstr, optional

Stratification dimension name (e.g., "stratum_ks").

Returns:
xr.Dataset

Dataset containing the calculated detailed weight proportions, indexed by all grouping and binning dimensions present in the input data.

Examples

>>> props = fitted_weight_proportions(
...     weight_data=da_scaled_unaged_weights,
...     aged_weight_proportions=ds_da_weight_proportion["aged"],
...     number_proportions=dict_ds_number_proportion["unaged"],
...     binned_weights=da_binned_weights_all,
...     stratum_dim="stratum_ks"
... )
>>> print(props)
<xarray.DataArray ...>
echopop.survey.proportions.fitted_weight_proportions_combined(number_proportions: Dataset, binned_weights: DataArray, stratum_dim: str | None = None) → Dataset#

Calculate weight proportions based on number proportions.

This function is used in workflows where the aged and unaged samples are pooled, such as when net selectivity is corrected based on length. The weight proportions are calculated by multiplying the number of proportions over sex, length, and aged with sex-specific mean weight at length, and then normalizing over all dimensions within a given stratum.

Parameters:
number_proportionsxr.Dataset

Number proportions that must contain a proportion variable with dimensions including stratum_dim, length_bin, and sex.

binned_weightsxr.DataArray

Mean fitted length-binned weights that must contain length_bin as a dimension and sex as a coordinate.

stratum_dimstr, optional

Stratification dimension name for normalizing proportions within each stratum (e.g., "stratum_ks").

Returns:
xr.Dataset

DataArray containing the calculated detailed weight proportions, indexed by all grouping and binning dimensions present in the input data.

Examples

>>> weight_props = fitted_weight_proportions_combined(
...     number_proportions=count_proportions,
...     binned_weights=da_binned_weights_all,
...     stratum_dim="stratum_ks",
... )
echopop.survey.proportions.get_nasc_proportions_slice(number_proportions: Dataset, ts_length_regression_parameters: dict[str, float], include_filter: dict[str, Any] | None = None, exclude_filter: dict[str, Any] | None = None, stratum_dim: str | None = None) → DataArray#

Extract and aggregate NASC proportions for specified groups from an xarray.DataArray.

This function selects and aggregates NASC proportions from an xarray.DataArray according to user-specified stratification dimension and filters, and returns the result as an xarray.DataArray. All grouping and filtering logic is handled dynamically based on the provided arguments.

Parameters:
number_proportionsxr.Dataset

xarray.Dataset of number proportions by relevant grouping factors, with dimensions including stratum_dim and binning columns (e.g., ‘sex’).

exclude_filterDict[str, Any], optional

Dictionary specifying values to exclude for filtering.

include_filterDict[str, Any], optional

Dictionary specifying values to include for filtering.

ts_length_regression_parametersDict[str, float]

Target strength-length regression parameters: - slope: regression slope - intercept: regression intercept

stratum_dimstr, optional

Stratification dimension name to use for grouping (e.g., "stratum_ks").

Returns:
xr.DataArray

DataArray containing the aggregated NASC proportions for each group/stratum, indexed by stratum_dim.

Notes

  • No dimension names are hard-coded; all grouping and filtering is dynamic.

  • The function assumes that grouping and binning dimensions are consistent with the DataArray structure.

  • Filtering is applied dynamically using include_filter and exclude_filter.

  • Output is an xarray.DataArray indexed by stratum_dim.

Examples

>>> props = get_nasc_proportions_slice(
...     number_proportions=aged_data,
...     stratum_dim="stratum_ks",
...     include_filter={"age_bin": [1]}
... )
>>> print(props)
<xarray.DataArray ...>
echopop.survey.proportions.get_number_proportions_slice(number_proportions: DataArray, stratum_dim: list[str] | None = None, exclude_filter: dict[str, Any] | None = None, include_filter: dict[str, Any] | None = None) → DataArray#

Extract and aggregate number proportions for specified groups from an xarray.DataArray.

This function selects and aggregates number proportions from an xarray.DataArray according to

Extract and aggregate number proportions for specified groups from an xarray.DataArray.

This function selects and aggregates number proportions from an xarray.DataArray according to user-specified grouping columns and filters, and returns the result as an xarray.DataArray. All grouping and filtering logic is handled dynamically based on the provided arguments.

Parameters:
number_proportionsxr.DataArray

xarray.DataArray of number proportions by relevant grouping factors, with dimensions including stratum_dim and binning columns (e.g., ‘length_bin’).

stratum_dimstr, optional

Stratification dimension name (e.g., "stratum_ks").

exclude_filterDict[str, Any], optional

Dictionary specifying values to exclude for filtering.

include_filterDict[str, Any], optional

Dictionary specifying values to include for filtering.

Returns:
xr.DataArray

DataArray containing the aggregated number proportions for each group/stratum, indexed by stratum_dim.

Notes

  • No dimension names are hard-coded; all grouping and filtering is dynamic.

  • The function assumes that grouping and binning dimensions are consistent with the DataArray structure.

  • Filtering is applied dynamically using include_filter and exclude_filter.

  • Output is an xarray.DataArray indexed by stratum_dim.

Examples

>>> props = get_number_proportions_slice(
...     number_proportions=dict_ds_number_proportion["aged"],
...     stratum_dim="stratum_ks",
...     include_filter={"age_bin": [1]}
... )
>>> print(props)
<xarray.DataArray ...>
echopop.survey.proportions.get_weight_proportions_slice(weight_proportions: Dataset, stratum_dim: str | None = None, include_filter: dict[str, Any] | None = None, exclude_filter: dict[str, Any] | None = None, number_proportions: dict[str, Dataset] | None = None, length_threshold_min: float | None = None, weight_proportion_threshold: float = 1e-10) → DataArray#

Calculate weight proportions for a specific population slice using xarray.

This function computes weight proportions for a target population group from an xarray.Dataset, with optional thresholding based on number proportions. Thresholding helps identify and mask unreliable estimates due to low sample sizes or small proportions. Number proportions must be provided as a dictionary of xarray.Dataset objects, each containing a "proportion" variable.

Parameters:
weight_proportionsxr.Dataset

Dataset with weight proportions and all relevant grouping dimensions.

stratum_dimstr, optional

Stratification dimension name (e.g., "stratum_ks").

include_filterDict[str, Any], default {}

Dictionary of filter criteria for the target group (e.g., {“age_bin”: [1]}).

exclude_filterDict[str, Any], default {}

Dictionary of groups to exclude (e.g., {“length_bin”: [small_lengths]}).

number_proportionsDict[str, xr.Dataset], default {}

Dictionary of xarray.Dataset objects for thresholding. Each Dataset must contain a “proportion” variable and relevant grouping/binning dimensions.

length_threshold_minfloat or None, default None

Minimum length for threshold calculations (e.g., 10.0 cm). Used only if number_proportions are provided. If None, no length-based thresholding is applied.

weight_proportion_thresholdfloat, default 1e-10

Threshold value for masking small proportions.

Returns:
xr.DataArray

Weight proportions by stratum_dim for the target group, with unreliable estimates masked to 0.

Notes

Thresholding Logic:

  1. Calculate target group weight proportions.

  2. If number_proportions are provided, calculate corresponding number thresholds.

  3. If length_threshold_min is set, exclude length bins below this threshold.

  4. Apply a mask where both weight and number proportions are very small.

  5. Set masked values to 0.0 to indicate unreliable estimates.

Examples

>>> weight_props = get_weight_proportions_slice(
...     weight_proportions,
...     stratum_dim="stratum_ks",
...     include_filter={"age_bin": [1]},
...     number_proportions={"aged": ds_aged, "unaged": ds_unaged},
...     length_threshold_min=10.0
... )
>>> print(weight_props)
<xarray.DataArray ...>

FEAT apportionment functions for redistributing and standardizing population estimates.

Provides utilities for partitioning NASC, abundance, and biomass transect values across biological groups, reallocating excluded age/size classes, and distributing unaged fish counts using aged reference proportions.

echopop.survey.apportionment.distribute_population_estimates(data: DataFrame, proportions: dict[str, Dataset] | Dataset, variable: str, group_columns: list[str] = None, data_proportions_link: dict[str, str] | None = None) → DataArray | dict[str, DataArray]#

Distribution population estimates across bins.

Distribute population estimates (e.g. abundance, biomass) using proportions grouped by metrics such as age and length bins. This function takes population estimates and distributes them across different biological categories (e.g. sex, age, length) using calculated proportions. The distribution is performed by multiplying the population estimates with the corresponding proportions.

Parameters:
datapd.DataFrame

DataFrame containing population estimates. Must contain the columns specified by ‘variable’ and ‘group_columns’. If ‘data_proportions_link’ is defined, then the key value must also be present in the DataFrame.

proportionsUnion[Dict[str, xr.Dataset], xr.Dataset]

Proportion data for distributing the population estimates. Can be a single xarray.Dataset or a dictionary of Datasets (e.g., {“aged”: ds_aged, “unaged”: ds_unaged}). Each Dataset must contain a variable named “proportion”.

variablestr

Name of the column in data containing the values to distribute (e.g. “biomass”, “abundance”).

group_columnsList[str]

List of column names that define the biological groups for distribution and stratification (e.g. [“sex”, “age_bin”, “length_bin”, “stratum_ks”]).

data_proportions_linkOptional[Dict[str, str]], default None

Dictionary mapping column names from ‘data’ to those in ‘proportions’. For instance, the dictionary {‘stratum_A’: ‘stratum_B’} links ‘stratum_A’ within ‘data’ with ‘stratum_B’ in the ‘proportions’ Dataset(s).

Returns:
Union[xr.DataArray, Dict[str, xr.DataArray]]

Distributed estimates. If a single group is present, returns an xarray.DataArray. Otherwise, returns a dictionary of xarray.DataArray objects, each containing the original biological group structure with values distributed according to proportions.

Notes

The DataArrays in proportions must have dimensions that correspond to columns in the data. Missing indices will result in NaN values for those rows. The function automatically identifies and uses the appropriate columns for merging and grouping.

echopop.survey.apportionment.distribute_unaged_from_aged(population_table: DataArray, reference_table: DataArray, stratum_dim: str | None = None, impute: bool = True, impute_variable: list[str] | None = None) → DataArray#

Standardize and optionally impute population estimates using reference proportions.

This function redistributes population estimates (e.g., unaged fish) to match the age/size structure observed in a reference table (e.g., aged fish), ensuring consistency across biological groups. Standardization is performed by summing over the stratification dimension in stratum_dim, then scaling by the corresponding reference proportions. Optionally, nearest-neighbor imputation is performed for missing or zero-valued intervals.

Parameters:
population_tablexr.DataArray

The input population estimates to be standardized. Must have coordinates for stratum_dim and typically for “length_bin”.

reference_tablexr.DataArray

The reference population data used for standardization. Should have matching or compatible coordinates/dimensions as population_table, with additional detail (e.g., age information).

stratum_dimstr, optional

Stratification dimension name to collapse (sum over) during standardization (e.g., "stratum_ks"). If None, no collapsing is performed.

imputebool, default True

If True, perform nearest-neighbor imputation for missing/zero values after standardization.

impute_variableList[str], optional

List of dimension names along which imputation is performed (e.g., [“age_bin”]). Required if impute=True.

Returns:
xr.DataArray

Standardized (and optionally imputed) population estimates, with the same structure as population_table but adjusted according to reference proportions.

Raises:
ValueError

If required coordinates (e.g., “length_bin”) are missing from either input table. If impute=True and impute_variable is not provided.

Notes

  • stratum_dim is the dimension that will be summed over (collapsed) in both population_table and reference_table. For example, if stratum_dim="stratum_ks", the result will sum over the “stratum_ks” dimension, producing a table without it.

  • The function expects both population_table and reference_table to have compatible coordinates and dimensions.

  • If impute=True, missing or zero-valued slices are filled using nearest-neighbor imputation.

  • The function validates that required coordinates (e.g., “length_bin”) are present and raises an error if not.

Examples

>>> result = distribute_unaged_from_aged(
...     population_table=da_unaged,
...     reference_table=da_aged,
...     stratum_dim="stratum_ks",
...     impute=True,
...     impute_variable=["age_bin"]
... )
>>> print(result)
<xarray.DataArray ...>
echopop.survey.apportionment.impute_kriged_table(initial_table: DataArray, reference_table: DataArray, standardized_table: DataArray, group_columns: list[str], subgroup_coords: list[str], impute_variable: list[str]) → DataArray#

Impute missing slices in binned kriged population estimates.

Impute missing or zero-valued slices in a standardized xarray DataArray using reference and initial tables. This function identifies zero-valued intervals in the reference table for grouped estimates, finds the nearest nonzero reference interval and imputes values in the standardized table using a ratio-based approach. The imputation is performed only where the initial table is nonzero and the reference table is zero, and is done for each group and interval independently.

Parameters:
initial_tablexr.DataArray

The initial (unstandardized) table, typically representing population or abundance estimates.

reference_tablexr.DataArray

The reference table used to guide imputation, typically representing a more complete or trusted set of estimates.

standardized_tablexr.DataArray

The standardized table to be imputed, which will be updated and returned.

group_columnsList[str]

List of dimension names used to group the data (e.g., [“stratum”]).

subgroup_coordsList[str]

List of coordinate names that define subgroups (typically a subset of group_columns).

impute_variableList[str]

List of dimension names along which imputation is performed (e.g., [“age_bin”]).

Returns:
xr.DataArray

The imputed standardized table, with missing or zero-valued slices replaced by imputed values.

Raises:
ValueError

If imputation fails for any group and interval combination.

Notes

The imputation is performed as follows:

  • For each group, identify intervals in the reference table that are zero but nonzero in the

    initial table.

  • For each such interval, find the nearest nonzero interval in the reference table.

  • Impute values in the standardized table using the formula:

    imputed = initial[interval] * sum referenced[impute variable x interval] /
    summed reference[interval]

    where the reference values are taken from the nearest nonzero interval.

  • The function updates the standardized table in-place and returns it.

echopop.survey.apportionment.mesh_biomass_to_nasc(mesh_data: DataFrame, biodata: DataFrame | dict[str, DataFrame], mesh_biodata_link: dict[str, str], stratum_weights: DataFrame, stratum_sigma_bs: DataFrame, group_columns: list[str] = None) → None#

Convert kriged biomass estimates into abundance and NASC.

Convert biomass estimates distributed across a grid or mesh into grouped (e.g. sex-specific) abundance and NASC values. This function takes kriged biomass density estimates from a mesh/grid and converts them to abundance and NASC (Nautical Area Scattering Coefficient) values by applying biological proportion data and acoustic backscattering coefficients. The function modifies the input mesh DataFrame in place.

Parameters:
mesh_datapd.DataFrame

DataFrame containing kriged biomass density estimates with spatial mesh information. Must contain columns for linking to biodata and either ‘biomass’, or ‘biomass_density’ and ‘area’, columns.

biodatapd.DataFrame or Dict[str, pd.DataFrame]

Biological proportion data. Can be a single DataFrame or dictionary of DataFrames containing proportion data for different biological groups (e.g., aged/unaged).

mesh_biodata_linkDict[str, str]

Dictionary mapping column names from mesh_data to biodata column names for joining. Example: {“geostratum_ks”: “stratum_ks”}

stratum_weightspd.DataFrame

DataFrame containing average weights per stratum for converting biomass to abundance. Index should match the linked columns from mesh_biodata_link.

stratum_sigma_bspd.DataFrame

DataFrame containing sigma_bs (backscattering coefficient) values per stratum. Must have ‘sigma_bs’ column with index matching linked columns.

group_byList[str], default []

List of column names to group biological data by (e.g., [“sex”, “age_bin”]).

Returns:
None

Function modifies mesh_data in place, adding new columns for biomass, abundance, and NASC values.

Raises:
KeyError

If required link columns are missing from biodata or mesh DataFrames.

Notes

The function performs the following steps:

  1. Computes biomass from biomass_density x area if not present

  2. Links mesh data to biological proportions using mesh_biodata_link

  3. Applies proportions to distribute total biomass across biological groups

  4. Converts biomass to abundance using stratum weights

  5. Calculates NASC using abundance and sigma_bs values

The NASC calculation uses the formula: NASC = abundance x sigma_bs x 4π

echopop.survey.apportionment.reallocate_excluded_estimates(population_table: DataArray, exclusion_filter: dict[str, Any], group_columns: list[str] = None) → DataArray#

Redistribute population estimates after excluding specified segments.

Removes population segments matching the exclusion filter and proportionally reallocates their values across the remaining segments, preserving totals within each group defined by group_columns. This is useful for excluding specific categories (e.g., age-1 fish) while maintaining total population estimates within groups.

Parameters:
population_tablexr.DataArray

Population estimates. Must have all dimensions referenced in exclusion_filter and group_columns.

exclusion_filterDict[str, Any]

Dictionary specifying which segments to exclude. Keys are dimension names, values are categories to exclude. Example: {“age_bin”: [1]} excludes age-1 fish.

group_columnsList[str], optional

Dimensions to group by when redistributing excluded values (e.g., [“sex”]). If empty, redistribution is performed over all dimensions.

Returns:
xr.DataArray

Population table with excluded segments set to zero and their values redistributed across remaining segments.

Raises:
ValueError

If any exclusion_filter or group_columns dimension is missing from population_table.

UserWarning

If the sum of the redistributed table does not match the original within a small tolerance.

Notes

  • Excluded segments are set to zero.

  • Their totals are redistributed proportionally across remaining segments within each group.

  • The function preserves the total population within each group defined by group_columns.

Examples

>>> result = reallocate_excluded_estimates(
...     population_table=da_population,
...     exclusion_filter={"age_bin": [1]},
...     group_columns=["sex"]
... )
>>> print(result)
<xarray.DataArray ...>
echopop.survey.apportionment.remove_group_from_estimates(transect_data: DataFrame, group_proportions: Dataset) → DataFrame#

Partition along-transect population estimates.

Partition NASC, abundance (and number density), and biomass (and biomass density) transect values across indexed groups.

Parameters:
transect_datapd.DataFrame

DataFrame containing transect data with number densities already computed. Must include: - A column that matches the index of the group_proportions DataArrays.

group_proportionsxr.Dataset

xarray.Dataset containing partitioning data for NASC, abundance, and biomass. Valid variable names are limited to ‘abundance’, ‘biomass’, and ‘nasc’. Each variable must be a 1D DataArray with dimensions matching grouping columns in transect_data.

Returns:
pd.DataFrame

DataFrame with the same structure as the input dataset, but with defined partitions applied to NASC and the other abundance/biomass columns.

Notes

The DataArrays in group_proportions must have dimensions that correspond to columns in the transect_data. Missing indices will result in NaN values for those rows. The function automatically identifies and uses the appropriate columns for merging.

echopop.survey.apportionment.sum_population_tables(population_tables: dict[str, DataArray]) → DataArray#

Combine and sum population estimates from multiple input tables into a single result.

This function takes a dictionary of population estimate tables and produces a single table by summing over any dimensions that are not shared by all input tables. Only dimensions present in every input table are retained in the result.

Parameters:
population_tablesDict[str, xr.DataArray]

Dictionary of population estimate tables to combine. Keys are table names and values are population tables with compatible dimensions.

Returns:
xr.DataArray

Combined population table with only the shared dimensions retained. Values are summed across all input tables for each combination of shared dimensions.

Notes

  • Any dimension not present in all input tables is summed over before combining.

  • The result is aligned on shared dimensions and summed element-wise.

  • The function is robust to input tables with different sets of dimensions, as long as there is at least one shared dimension.

Examples

>>> combined = sum_population_tables({
...     "aged": table_aged,
...     "unaged": table_unaged
... })
>>> print(combined)
<xarray.DataArray ...>

Net selectivity correction#

echopop.survey.selectivity.assign_selectivity_expansion(biodata: DataFrame, net_selectivity_params: dict[str, float] | dict[str, dict[str, float]], net_column: str, minimum_selectivity: float = 1e-12) → DataFrame#

Compute per-fish logistic selectivity expansion factors and add them as a new column.

The caller is responsible for ensuring biodata already contains both a length column and a gear column (e.g., via a prior merge of the individual-level and haul-level frames). This function only performs the selectivity computation.

Parameters:
biodatapd.DataFrame

Individual-level DataFrame containing at least net_column and ‘length’.

net_selectivity_paramsdict

A dictionary defining the selectivity parameters. This can take two shapes:

1. Global (single parameter set)

Applied uniformly to every row in biodata. Example: {'l50': 12.5, 'sr': 3.0}

2. Gear-Specific (multiple net-types with unique parameters)

A mapping of net types to their respective parameter dictionaries. Example:

{
    'AWT': {'l50': 15.0, 'sr': 4.5},
    'IKMT': {'intercept': -10.2, 'slope': 0.8}
}

Each parameter set must consist of either selection parameters ('l50', 'sr') or logistic regression coefficients ('slope', 'intercept'). When a row’s net-type is not found in net_selectivity_params, its selectivity expansion (\(S(L)\)) will be NaN.

net_columnstr

Column identifying the net-type for each individual.

minimum_selectivityfloat, default 1e-12

Lower bound applied to S(L) before inversion to prevent division by zero.

Returns:
pd.DataFrame

Copy of biodata with the following columns added:

  • l50 : The length-at-50%-retention applied to that row.

  • sr : The selection range applied to that row.

  • selectivity_expansion : The calculated expansion factor (\(\\phi(L)\)).

Raises:
KeyError

If net_column or ‘length’ are missing from biodata.

ValueError

If the parameter dictionary contains mixed or incomplete sets.

Notes

The probability of retention \(S(L)\) for a fish of length \(L\) is modeled using the logistic selection ogive:

\[\begin{split}S(L) = \\left[ 1 + \\exp\\left( \\frac{2\\ln(3)(L_{50} - L)}{SR} \\right) \\right]^{-1}\end{split}\]

If the user provides logistic regression coefficients (intercept \(\\alpha\) and slope \(\\beta\)), they are converted to selection parameters via:

\[\begin{split}L_{50} = -\\frac{\\alpha}{\\beta}, \\quad SR = \\frac{2\\ln(3)}{\\beta}\end{split}\]

The resulting expansion factor \(\\phi(L)\) is the inverse of the retention probability, capped by a minimum threshold \(S_{\\min}\):

\[\begin{split}\\phi(L) = \\frac{1}{\\max(S(L), S_{\\min})},\end{split}\]

where \(S_{\\min}\) is the minimum selectivity used for computational purposes.

References

Wileman, D. A., Ferro, R. S. T., Fonteyne, R., & Millar, R. B. (Eds.). (1996). Manual of methods of measuring the selectivity of towed fishing gears. ICES Cooperative Research Report No. 215.

Statistics#

echopop.survey.statistics.confidence_interval(bootstrap_samples: Series | DataFrame, population_values: Series | DataFrame, ci_method: Literal['bc', 'bca', 'empirical', 'normal', 'percentile', 't', 't-jackknife'], ci_percentile: float = 0.95)#

Calculate confidence intervals for bootstrap samples using various methods.

This function provides multiple methods for computing confidence intervals from bootstrap replicates, including bias-corrected methods and studentized approaches.

Parameters:
bootstrap_samplesUnion[pd.Series, pd.DataFrame]

DataFrame or Series containing bootstrap replicates. Each column represents a different variable, and each row represents a bootstrap replicate.

population_valuesUnion[pd.Series, pd.DataFrame]

DataFrame or Series containing the original (non-bootstrap) population statistics for each variable. Must have the same column structure as bootstrap_samples.

ci_method{“bc”, “bca”, “empirical”, “normal”, “percentile”, “t”, “t-jackknife”}

Method for computing the bootstrap confidence interval:

  • “bc”: Bias-corrected method[Rf8b45216ba75-1]_

  • “bca”: Bias-corrected and accelerated method[Rf8b45216ba75-2]_

  • “empirical”: Empirical method using bootstrap deviations[Rf8b45216ba75-1]_

  • “normal”: Normal approximation[Rf8b45216ba75-3]_

  • “percentile”: Simple percentile method[Rf8b45216ba75-1]_

  • “t”: t-distribution based[Rf8b45216ba75-4]_

  • “t-jackknife”: Jackknife studentized method[Rf8b45216ba75-4]_

ci_percentilefloat, default 0.95

Confidence level (between 0 and 1). For example, 0.95 gives a 95% CI.

Returns:
pd.DataFrame

DataFrame containing confidence interval results with the following structure:

  • If population_values has additional column index levels: Returns a DataFrame with confidence interval bounds and statistics stacked appropriately

  • If population_values has simple columns: Returns a transposed DataFrame with metrics as columns

Raises:
TypeError

If population_values is not a pandas DataFrame or Series.

ValueError

If ci_method is not one of the supported methods.

KeyError

If bootstrap_samples and population_values don’t have matching columns.

Notes

The function automatically handles different DataFrame structures and provides appropriate output formatting. Bias calculations are included in the output for all methods.

References

[1]

Efron, B. (1981). Nonparametric standard errors and confidence intervals. Canadian Journal of Statistics, 9(2), 139-158.

[2]

Efron, B., and Tibshirani, R.J. (1993). An Introduction to the Bootstrap. Springer US. https://doi.org/10.1007/978-1-4899-4541-9

[3]

Efron, B., and Tibshirani, R.J. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science, 1(1), 54-75.

[4]

DiCiccio, T.J., and Efron, B. (1996). Bootstrap confidence intervals. Statistical Science, 11(3), 189-228.

Examples

>>> import pandas as pd
>>> import numpy as np
>>> bootstrap_data = pd.DataFrame({
...     'biomass': np.random.normal(100, 10, 1000),
...     'density': np.random.normal(50, 5, 1000)
... })
>>> population_data = pd.DataFrame({
...     'biomass': [95],
...     'density': [48]
... })
>>> ci_results = confidence_interval(
...     bootstrap_data, population_data,
...     ci_method='percentile', ci_percentile=0.95
... )
class echopop.survey.stratified.JollyHampton(model_parameters: dict[str, Any], resample_seed: int | None = None)#

Jolly-Hampton stratified survey analysis with bootstrap resampling.

This class implements stratified sampling analysis using the Jolly-Hampton method with bootstrap resampling for uncertainty estimation. It handles virtual transect creation, stratified bootstrap sampling, and confidence interval estimation.

Parameters:
model_parametersDict[str, Any]

Dictionary containing model configuration parameters:

  • “transects_per_latitude”: Number of transects per degree latitude

  • “strata_transect_proportion”: Proportion of transects to sample per stratum

  • “num_replicates”: Number of bootstrap replicates

resample_seedint, optional

Random seed for reproducible bootstrap resampling, by default None.

Attributes:
model_paramsDict[str, Any]

Stored model parameters.

rngnp.random.Generator

Random number generator for bootstrap sampling.

bootstrap_replicatespd.DataFrame or None

DataFrame containing bootstrap replicates (set after stratified_bootstrap).

variablestr or None

Name of the response variable being analyzed.

transect_summarypd.DataFrame

Summary statistics for each transect within strata.

strata_summarypd.DataFrame

Summary statistics for each stratum.

survey_summaryDict[str, pd.DataFrame]

Dictionary containing survey-level summaries for strata and overall survey.

Methods

create_virtual_transects(mesh_data, ...)

Create virtual transects from gridded data and assign to strata.

stratified_bootstrap(data, stratum_dim, variable)

Perform stratified bootstrap resampling analysis.

summarize([ci_percentile, ci_method])

Generate summary statistics and confidence intervals from bootstrap results.

Notes

The Jolly and Hampton algorithm is commonly used in fisheries acoustic surveys for estimating fish biomass and abundance with stratified sampling designs.

References

Jolly, G.M., and Hampton, I. (1990). A stratified random transect design for acoustic surveys of fish stocks. Canadian Journal of Fisheries and Aquatic Sciences, 47(7), 1282-1291. https://doi.org/10.1139/f90-147

Examples

>>> model_params = {
...     "transects_per_latitude": 5,
...     "strata_transect_proportion": 0.75,
...     "num_replicates": 100,
... }
>>> analysis = JollyHampton(model_params, resample_seed=42)
>>> virtual_data = analysis.create_virtual_transects(
...     mesh_data, geostratum_df,
...     stratify_by=["geostratum_inpfc"],
...     variable="biomass"
... )
>>> analysis.stratified_bootstrap(
...     virtual_data,
...     stratify_by=["geostratum_inpfc"],
...     variable="biomass"
... )
>>> summary = analysis.summarize(ci_percentile=0.95)
create_virtual_transects(mesh_data: DataFrame, geostrata: DataFrame, stratum_dim: str, variable: str) → DataFrame#

Create virtual transects from gridded data and assign to strata.

This method converts gridded data into virtual transects by partitioning based on latitude, computing transect metrics, and assigning transects to geographical strata.

Parameters:
mesh_datapd.DataFrame

Input DataFrame containing gridded survey data with columns: latitude, longitude, area, and the response variable.

geostratapd.DataFrame

DataFrame containing geographical stratum boundaries and definitions.

stratum_dumstr

Stratification column name (typically geographical strata).

variablestr

Name of the response variable column (e.g., ‘biomass’, ‘abundance’).

Returns:
pd.DataFrame

DataFrame containing virtual transects with computed distances, areas, response variable sums, and stratum assignments.

Notes

The virtual transects represent aggregated data along latitude bands, which is a common approach in systematic acoustic surveys.

stratified_bootstrap(data: DataFrame, stratum_dim: str, variable: str) → None#

Perform stratified bootstrap resampling analysis.

Executes the complete bootstrap analysis pipeline including transect summarization, stratum summarization, bootstrap sample generation, and computation of bootstrap statistics.

Parameters:
datapd.DataFrame

Input DataFrame containing virtual transect data with all required columns for analysis.

stratum_dimstr

Column name defining the stratification (e.g., ‘geostratum_inpfc’).

variablestr

Name of the response variable column (e.g., ‘biomass’, ‘abundance’).

Notes

This method populates the ‘bootstrap_replicates’ attribute with bootstrap replicates organized by stratum and replicate. The bootstrap sampling is done without replacement within each stratum.

The method performs the following steps:

  1. Summarize transect-level data

  2. Summarize stratum-level data

  3. Generate bootstrap sample arrays

  4. Compute distance-weighted means and variances

  5. Format results into structured DataFrame

summarize(ci_percentile: float = 0.95, ci_method: Literal['bc', 'bca', 'empirical', 'normal', 'percentile', 't', 't-jackknife'] = 't-jackknife') → DataFrame#

Generate summary statistics and confidence intervals from bootstrap results.

Computes confidence intervals for both stratum-level and survey-wide estimates using the specified bootstrap method.

Parameters:
ci_percentilefloat, default 0.95

Confidence level for interval estimation (between 0 and 1).

ci_method{“bc”, “bca”, “empirical”, “normal”, “percentile”, “t”, “t-jackknife”},
default “t-jackknife”

Bootstrap confidence interval method:

  • “bc”: Bias-corrected

  • “bca”: Bias-corrected and accelerated

  • “empirical”: Empirical bootstrap

  • “normal”: Normal approximation

  • “percentile”: Simple percentile method

  • “t”: t-distribution based

  • “t-jackknife”: Jackknife studentized (recommended)

Returns:
pd.DataFrame

DataFrame containing confidence intervals and summary statistics for both stratum-level and survey-wide estimates (including the coefficient of variation). Includes columns for confidence bounds, means, and bias estimates. These quantities are calculated using the area-weighted stratified approach.

Raises:
RuntimeError

If stratified_bootstrap() has not been run prior to calling this method.

Notes

The method computes area-weighted survey statistics and includes coefficient of variation estimates. The t-jackknife method is recommended as it provides better coverage properties for small sample sizes common in survey data.

Examples

>>> summary_stats = analysis.summarize(
...     ci_percentile=0.95,
...     ci_method='t-jackknife'
... )
>>> print(summary_stats)

Transect processing#

echopop.survey.transect.compute_interval_distance(nasc_data: DataFrame, interval_threshold: float = 0.05) → None#

Calculate along-transect interval distances and add to DataFrame.

Parameters:
nasc_datapd.DataFrame

DataFrame containing NASC data with distance and spacing information. Must contain columns: ‘distance_s’, ‘distance_e’, ‘transect_spacing’

interval_thresholdfloat, default 0.05

Along-transect interval threshold for detecting erroneous values. Values that deviate from the median interval by more than this threshold will be corrected using ‘distance_e’ - ‘distance_s’ calculation.

Returns:
pd.DataFrame

Modified DataFrame with added ‘distance_interval’ column containing along-transect interval distances.

Notes

This function calculates the along-track transect interval length. It identifies and corrects potentially erroneous values at transect endpoints by comparing intervals to the median and replacing outliers with direct distance calculations (distance_e - distance_s).

The interval calculation uses diff(periods=-1) to compute forward differences, making each interval represent the distance to the next measurement point.

Examples

>>> df = pd.DataFrame({
...     'distance_s': [0, 1, 2, 3],
...     'distance_e': [1, 2, 3, 4],
...     'transect_spacing': [0.1, 0.1, 0.1, 0.1]
... })
>>> set_interval_distance(df)
>>> 'distance_interval' in df.columns
True