Biological data ingestion#
Import necessary modules#
First, import the biological module from the ingest sub-package of latest version of Echopop for loading in the data.
from echopop.ingest import biological as load_data
Ingesting the data#
It is expected that all of the biodata are sourced from a single master spreadsheet with three specific sheets: 1) catch, 2) length, and 3) specimen. However, those exact sheetnames may not be present within the file, so a column_name_map needs to be made that links the expected types with their respective names.
BIODATA_SHEET_MAP = {
"catch": "biodata_catch",
"length": "biodata_length",
"specimen": "biodata_specimen",
}
The data from each sheet also need to be filtered for specific target ship identifiers, survey names, whether an offset is added to the haul numbering to avoid overlap, and species codes. Moreover, there may be columns in the file that do not match those expected by Echopop. Similarly, certain biodata labels like "sex" may need modification. These can be defined via:
CAN_HAUL_OFFSET = 200
SPECIES_ID = 22500 # numeric species code for Pacific hake
SHIP_US = 160 # US ship ID
SHIP_CAN = 584 # CAN ship ID
SURVEY_US = 201906 # US survey identifier
SURVEY_CAN = 2019097 # CAN survey identifier
BIODATA_SHIP_SPECIES = {
"ships": {
SHIP_US: {
"country": "US",
"survey": SURVEY_US
},
SHIP_CAN: {
"country": "CAN",
"survey": SURVEY_CAN,
"haul_offset": CAN_HAUL_OFFSET
}
},
"species_code": [SPECIES_ID]
}
EXPECTED_ECHOPOP_BIODATA_COLUMNS = {
"frequency": "length_count",
"haul": "haul_num",
"weight_in_haul": "weight",
}
BIODATA_LABEL_MAP = {
"sex": {
1: "male",
2: "female",
3: "unsexed"
}
}
One additional dictionary needs to be defined to configure the haul-based unique identifier, uid. This identifier strings together the ship ID, survey ID, species ID, and haul number via: ship_id-survey_id-species_id-haul_num. For instance, a specimen with:
ship_id = 41
survey_id = 2041
species_id = 22500
haul_num = 81
would have a uid of: 41-2041-22500-81.
This dictionary provides the requisite instructions for how to construct this identifier.
HAUL_UID_CONFIG = {
"ship_id": {"US": SHIP_US, "CAN": SHIP_CAN},
"survey_id": {"US": SURVEY_US, "CAN": SURVEY_CAN},
"species_id": SPECIES_ID,
"haul_offset": CAN_HAUL_OFFSET
}
All of this additional information can then be supplied to load_biological_data to read in the biological data file.
dict_df_bio = load_data.load_biological_data(
biodata_filepath=DATA_ROOT / "Biological/1995-2023_biodata_redo.xlsx",
biodata_sheet_map=BIODATA_SHEET_MAP,
column_name_map=EXPECTED_ECHOPOP_BIODATA_COLUMNS,
survey_subset=BIODATA_SHIP_SPECIES,
biodata_label_map=BIODATA_LABEL_MAP,
haul_uid_config=HAUL_UID_CONFIG,
)
Adding age and length bins#
Many of the analyses used by Echopop assume that the underlying biological datasets are distributed over length, or age and length. Consequently, our ingested biodata need to be “binified” here using the binify function from the utils module. Here we want to break up age into integer bins from 1 to 22 years at increments of 1 year, and length into float bins from 2 to 80 cm at increments of 2 cm.
import numpy as np
from echopop import utils
# Age bins
AGE_BINS = np.linspace(start=1., stop=22, num=22)
utils.binify(
data=dict_df_bio, bins=AGE_BINS, bin_column="age",
)
# Length bins
LENGTH_BINS = np.linspace(start=2., stop=80., num=40)
utils.binify(
data=dict_df_bio, bins=LENGTH_BINS, bin_column="length",
)
Fitting length-weight regressions#
The next step is to fit the length-weight regressions (Eq. 2.6) for each sex in the dataset, and then across all fish (inclusive of unsexed fish). This can be done using the biology module in the survey
from echopop.survey import fit_length_weight_regression
# Fit length-weight regression
# ---- Create dictionary container
dict_length_weight_coefs = {}
# ---- Regression for all fish
dict_length_weight_coefs["all"] = dict_df_bio["specimen"].assign(sex="all").groupby(["sex"]).apply(
fit_length_weight_regression,
include_groups=False
)
# ---- Regression for each sex
dict_length_weight_coefs["sex"] = dict_df_bio["specimen"].groupby(["sex"]).apply(
fit_length_weight_regression,
include_groups=False
)
Computing the mean weights per length bin#
The mean weights per length bin can be computed using the length_binned_weights function from the survey.biology module. Similar to the length-weight regression fitting, this can be done across all fish as well as for each sex individually.
import xarray as xr
from echopop.survey import biology
# Sex-specific (grouped coefficients)
da_binned_weights_sex = biology.length_binned_weights(
data=dict_df_bio["specimen"],
length_bins=LENGTH_BINS,
regression_coefficients=dict_length_weight_coefs["sex"],
impute_bins=True,
minimum_count_threshold=5
)
# All fish (single coefficient set)
da_binned_weights_all = biology.length_binned_weights(
data=dict_df_bio["specimen"].assign(sex="all"),
length_bins=LENGTH_BINS,
regression_coefficients=dict_length_weight_coefs["all"],
impute_bins=True,
minimum_count_threshold=5,
)
# Combine the pivot tables by adding the "all" column to the sex-specific table
da_binned_weight_table = xr.concat(
[da_binned_weights_sex, da_binned_weights_all],
dim = "sex"
)
This creates an xarray.DataArray object, which provides wraps an array of the fitted weights with labeled dimensions and coordinates. This structure eliminates the need to track axis indices (e.g., axis=0) manually like with a pandas.DataFrame. Furthermore, unlike a rectangular 2D table (like a standard spreadsheet of pandas.DataFrame), the fitted weights in da_binned_weight_table are stored in a multidimensional hypercube. In this dataset, the weights are mapped across two primary dimensions, but the structure is designed to scale:
Length bins (\(d_1\)): Represents the discrete length bins.
Sex (\(d_2\)): Represents animal sex.
<xarray.DataArray 'weight_fitted' (length_bin: 40, sex: 4)> Size: 1kB
array([[2.84964284e-05, 3.15793921e-05, 2.77646527e-03, 3.09226848e-05],
[2.58749425e-04, 2.79801027e-04, 6.40972110e-03, 2.75469178e-04],
[9.40427909e-04, 1.00246545e-03, 1.04564300e-02, 9.90074672e-04],
[2.34946163e-03, 2.47910455e-03, 1.47974207e-02, 2.45396764e-03],
[4.77973820e-03, 5.00384882e-03, 1.93712476e-02, 4.96174900e-03],
[8.53914663e-03, 8.88208560e-03, 2.41396141e-02, 8.81990220e-03],
[1.39471732e-02, 1.44284494e-02, 2.90760842e-02, 1.43446898e-02],
[2.13332645e-02, 2.19654640e-02, 3.41611834e-02, 2.18607295e-02],
[3.10356315e-02, 3.18225565e-02, 3.93799017e-02, 3.16999595e-02],
[4.96857143e-02, 5.08333333e-02, 4.47202763e-02, 5.02261905e-02],
[6.34794521e-02, 6.33150685e-02, 5.01725247e-02, 6.33741497e-02],
[7.97200000e-02, 7.95178571e-02, 5.57284817e-02, 7.96132075e-02],
[1.08000000e-01, 9.20000000e-02, 6.13812175e-02, 1.01454545e-01],
[1.37333333e-01, 1.31500000e-01, 6.71247691e-02, 1.34588235e-01],
[1.67666667e-01, 1.68000000e-01, 7.29539451e-02, 1.67833333e-01],
[2.01750000e-01, 1.96341463e-01, 7.88641804e-02, 1.99012346e-01],
[2.44032258e-01, 2.46977273e-01, 8.48514260e-02, 2.45760000e-01],
[3.10680000e-01, 3.10860000e-01, 9.09120634e-02, 3.10770000e-01],
[3.71516129e-01, 3.75314607e-01, 9.70428370e-02, 3.73754967e-01],
[4.30414013e-01, 4.27462451e-01, 1.03240801e-01, 4.28592683e-01],
[4.87911972e-01, 4.81691689e-01, 1.09503275e-01, 4.84380518e-01],
[5.48851986e-01, 5.37892256e-01, 1.15827809e-01, 5.43181185e-01],
[6.16933333e-01, 6.04986486e-01, 1.22212155e-01, 6.11778426e-01],
[6.99440367e-01, 6.80173913e-01, 1.28654238e-01, 6.91971910e-01],
[8.28064935e-01, 8.53290323e-01, 1.35152143e-01, 8.35305556e-01],
[9.26200000e-01, 9.43818182e-01, 1.41704090e-01, 9.29660714e-01],
[1.13100000e+00, 1.12900000e+00, 1.48308424e-01, 1.13063636e+00],
[1.28922727e+00, 1.28440000e+00, 1.54963598e-01, 1.28833333e+00],
[1.38109091e+00, 1.26495823e+00, 1.61668168e-01, 1.38266667e+00],
[1.69733333e+00, 1.40739294e+00, 1.68420778e-01, 1.69733333e+00],
[1.61620000e+00, 1.56039646e+00, 1.75220152e-01, 1.61620000e+00],
[1.75887608e+00, 1.72437397e+00, 1.82065091e-01, 1.73482746e+00],
[1.93985272e+00, 1.89973257e+00, 1.88954460e-01, 1.91170835e+00],
[2.13320417e+00, 2.08688125e+00, 1.95887188e-01, 2.10052633e+00],
[2.33937516e+00, 2.28623083e+00, 2.02862262e-01, 2.30170021e+00],
[2.55881277e+00, 2.49819390e+00, 2.09878718e-01, 2.51565073e+00],
[2.79196648e+00, 2.72318481e+00, 2.16935641e-01, 2.74280050e+00],
[3.03928800e+00, 2.96161960e+00, 2.24032163e-01, 2.98357394e+00],
[3.30123133e+00, 3.21391597e+00, 2.31167454e-01, 3.23839726e+00],
[3.57825261e+00, 3.48049325e+00, 2.38340723e-01, 3.50769842e+00]])
Coordinates:
* length_bin (length_bin) category 680B (1.0, 3.0] ... (79.0, 81.0]
* sex (sex) object 32B 'female' 'male' 'unsexed' 'all'