Configure workflow scripts

Configure workflow scripts#

Separating configuration from executable code enables users to define workflows without needing to constantly update the underlying source code. By providing parameters such as input directories, parameter values, and other instructions within an external text file, one can create a customized execution plan without the need for duplicating workflow files. This approach transforms the software into a versatile engine where the configuration file acts as the primary interface for defining what the workflow should execute.

The primary advantage of using external configuration files is the ability to maintain a clear, portable record of specific analysis and project settings. This confers three major benefits:

Reproducibility: Each configuration file serves as a documented history of the exact settings used for a specific run, which is essential for auditing and consistent results across different end-users.
Flexibility: Multiple versions of a configuration file can test different scenarios or process separate datasets without needing to recreate the entire workflow from scratch.
Accessibility: This design enables users to modify program behavior by simply editing a text file, which removes the requirement to navigate or understand every component of the underlying script structure.

Configuration formats#

Configuration files can be defined using a variety of file formats including JSON, YAML, TOML, INI, and even plain TXT. These structured data standards allow for the specification of operational parameters within a portable, text-based file that exists independently of the core Python package. Selecting the format that best aligns with specific technical requirements facilitates the creation of a clear and repeatable set of instructions for the software.

Choosing the right file format

Selection of the appropriate format depends on the complexity of the workflow and the primary method of interaction. For workflows requiring frequent manual updates, YAML is often preferred due to support for descriptive comments and a clean visual hierarchy. Conversely, if a configuration is generated or consumed by automated tools and web services, JSON provides a more rigid and widely compatible structure. Unstructured formats like TXT should be avoided unless the workflow requires only a single, simple value, as these files lack the built-in validation and nesting capabilities provided by established standards.

While various standards exist for data serialization, this guide focuses on JSON and YAML due to their extensive library support and common usage in Python-based workflow automation.

JSON#

JSON, or JavaScript Object Notation, is a lightweight data interchange format widely used for web services and automated systems. It utilizes a strict structure of brackets and braces to organize information into key-value pairs and lists.

Visual Style: Brackets and Braces
Comments: Not Supported
Data Relationships: Highly structured nested objects
Primary Use: APIs and automated data interchange

Due to this rigid syntax, JSON is highly efficient for machine processing and ensures that configuration data remains consistent across different platforms.

YAML#

YAML, or YAML Ain’t Markup Language, is a human-friendly data serialization standard designed for readability. It relies on indentation and whitespace rather than special characters to define the hierarchy of a workflow.

Visual Style: Indentation and Whitespace
Comments: Supported (using the # symbol)
Data Relationships: Visual hierarchy through indentation
Primary Use: Manual configurations and setup files

This format is particularly popular for manual configuration because it supports internal comments, allowing for the documentation of specific settings directly within the file.

Modifying workflows#

Say that we have the following example workflow in a file called example_biodata_ingestion_workflow.py. This section only targets the biodata setup and ingestion. The goal is to externalize the parameters that users change most often, while leaving the rest of the workflow untouched.

Show code cell source

Hide code cell source

# Import packages
from pathlib import Path
from echopop import ingest
from echopop.survey import biology

# Root data directory
DATA_ROOT = Path("C:/Data/EchopopData/")

# Biodata file [Excel]
BIODATA_FILE = DATA_ROOT / "survey_biodata.xlsx"

# Biodata file sheetnames
BIODATA_SHEETS = {
    "catch": "biodata_catch",
    "length": "biodata_length",
    "specimen": "biodata_specimen",
}

# Different biodata processing parameters required for parsing the Excel-formatted dataset
SPECIES_ID = 22500 # numeric species code for Pacific hake
SHIP_US = 160 # US ship ID
SHIP_CAN = 584 # CAN ship ID
SURVEY_US = 201906 # US survey identifier
SURVEY_CAN = 2019097 # CAN survey identifier
CAN_HAUL_OFFSET = 200
# ---- Dictionary structure of this applied
BIODATA_SHIP_SPECIES = {
    "ships": {
        SHIP_US: {
            "country": "US",
            "survey": SURVEY_US 
        },
        SHIP_CAN: {
            "country": "CAN",
            "survey": SURVEY_CAN,
            "haul_offset": CAN_HAUL_OFFSET
        }
    },
    "species_code": [SPECIES_ID]
}

# Haul UID reference dictionary
HAUL_UID_CONFIG = {
    "ship_id": {"US": SHIP_US, "CAN": SHIP_CAN},
    "survey_id": {"US": SURVEY_US, "CAN": SURVEY_CAN},
    "species_id": SPECIES_ID,
    "haul_offset": CAN_HAUL_OFFSET
}

# Biodata ingestion parameters
RENAME_BIODATA_COLUMNS = {
    "frequency": "length_count",
    "haul": "haul_num",
    "haul ": "haul_num",
    "weight_in_haul": "weight",
}

# Biodata sex label mapping
BIODATA_SEX = {
    "sex": {
        1: "male",
        2: "female",
        3: "unsexed"
    }
}

# Ingest biodata
dict_df_bio = ingest.load_biological_data(
    biodata_filepath=BIODATA_FILE, 
    biodata_sheet_map=BIODATA_SHEETS, 
    column_name_map=RENAME_BIODATA_COLUMNS, 
    survey_subset=BIODATA_SHIP_SPECIES, 
    biodata_label_map=BIODATA_SEX,
    haul_uid_config=HAUL_UID_CONFIG,
)

# Remove specimen-only hauls
biology.drop_specimen_only_hauls(dict_df_bio)

This block contains many values that users commonly edit between runs, including data paths, sheet mappings, ship and survey identifiers, species filters, and column mapping dictionaries. Moving those values into a configuration file reduces copy paste edits and makes each run definition easier to review and reproduce.

JSON

{
  "data_root": "C:/Data/EchopopData",
  "biodata_file": "survey_biodata.xlsx",
  "biodata_sheets": {
    "catch": "biodata_catch",
    "length": "biodata_length",
    "specimen": "biodata_specimen"
  },
  "biodata_subset": {
    "species_id": 22500,
    "ship_us": 160,
    "ship_can": 584,
    "survey_us": 201906,
    "survey_can": 2019097,
    "can_haul_offset": 200,
  },  
  "rename_biodata_columns": {
    "frequency": "length_count",
    "haul": "haul_num",
    "haul ": "haul_num",
    "weight_in_haul": "weight"
  }
}

YAML

data_root: C:/Data/EchopopData
biodata_file: survey_biodata.xlsx
biodata_sheets:
  catch: biodata_catch
  length: biodata_length
  specimen: biodata_specimen
biodata_subset:
  species_id: 22500
  ship_us: 160
  ship_can: 584
  survey_us: 201906
  survey_can: 2019097
  can_haul_offset: 200
rename_biodata_columns:
  frequency: length_count
  haul: haul_num
  "haul ": haul_num
  weight_in_haul: weight

Loading JSON and YAML into this workflow block#

For this explicit example, read the config file directly into config_params, then use config_params to populate the same variables used in the original snippet.

Read JSON directly

from pathlib import Path
import json

config_path = Path("configs/biodata_setup.json")
config_params = json.loads(config_path.read_text(encoding="utf-8"))

Read YAML directly

from pathlib import Path
import yaml

config_path = Path("configs/biodata_setup.yaml")
config_params = yaml.safe_load(config_path.read_text(encoding="utf-8"))

When this pattern is used in production, validate that required keys exist and raise a clear error before any data ingestion call is executed. This can be achieved using approaches ranging from if-else blocks to existing libraries like Pydantic.

Applying user configurations#

The previous biodata setup and ingestion code can be rewritten by incorporating the loaded configuration file. It changes only the parameter sourcing and keeps the ingestion logic intact.

Original code block

# Import packages
from pathlib import Path
from echopop import ingest
from echopop.survey import biology

# Root data directory
DATA_ROOT = Path("C:/Data/EchopopData/")

# Biodata file [Excel]
BIODATA_FILE = DATA_ROOT / "survey_biodata.xlsx"

# Biodata file sheetnames
BIODATA_SHEETS = {
    "catch": "biodata_catch",
    "length": "biodata_length",
    "specimen": "biodata_specimen",
}

# Different biodata processing parameters required for parsing the Excel-formatted dataset
SPECIES_ID = 22500 # numeric species code for Pacific hake
SHIP_US = 160 # US ship ID
SHIP_CAN = 584 # CAN ship ID
SURVEY_US = 201906 # US survey identifier
SURVEY_CAN = 2019097 # CAN survey identifier
# ---- Dictionary structure of this applied
BIODATA_SHIP_SPECIES = {
    "ships": {
        SHIP_US: {
            "country": "US",
            "survey": SURVEY_US 
        },
        SHIP_CAN: {
            "country": "CAN",
            "survey": SURVEY_CAN,
            "haul_offset": CAN_HAUL_OFFSET
        }
    },
    "species_code": [SPECIES_ID]
}

# Haul UID reference dictionary
HAUL_UID_CONFIG = {
    "ship_id": {"US": SHIP_US, "CAN": SHIP_CAN},
    "survey_id": {"US": SURVEY_US, "CAN": SURVEY_CAN},
    "species_id": SPECIES_ID,
    "haul_offset": CAN_HAUL_OFFSET
}

# Biodata ingestion parameters
RENAME_BIODATA_COLUMNS = {
    "frequency": "length_count",
    "haul": "haul_num",
    "haul ": "haul_num",
    "weight_in_haul": "weight",
}

# Biodata sex label mapping
BIODATA_SEX = {
    "sex": {
        1: "male",
        2: "female",
        3: "unsexed"
    }
}

# Ingest biodata
dict_df_bio = ingest.load_biological_data(
    biodata_filepath=BIODATA_FILE, 
    biodata_sheet_map=BIODATA_SHEETS, 
    column_name_map=RENAME_BIODATA_COLUMNS, 
    survey_subset=BIODATA_SHIP_SPECIES, 
    biodata_label_map=BIODATA_SEX,
    haul_uid_config=HAUL_UID_CONFIG,
)

# Remove specimen-only hauls
feat_biology.drop_specimen_only_hauls(dict_df_bio)

JSON configuration

# Import packages
from pathlib import Path
from echopop import ingest
import json
from echopop.survey import biology

# Read in configuration
config_path = Path("configs/biodata_setup.json")
config_params = json.loads(config_path.read_text(encoding="utf-8"))

# Root data directory
DATA_ROOT = Path(config_params["data_root"])

# Biodata file [Excel]
BIODATA_FILE = DATA_ROOT / config_params["biodata_file"]

# Biodata file sheetnames
BIODATA_SHEETS = config_params["biodata_sheets"]

# Different biodata processing parameters required for parsing the Excel-formatted dataset
BIODATA_SHIP_SPECIES = {
    "ships": {
        config_params["biodata_subset"]["ship_us"]: {
            "country": "US",
            "survey": config_params["biodata_subset"]["survey_us"] 
        },
        config_params["biodata_subset"]["ship_can"]: {
            "country": "CAN",
            "survey": config_params["biodata_subset"]["survey_can"],
            "haul_offset": config_params["biodata_subset"]["can_haul_offset"]
        }
    },
    "species_code": [config_params["biodata_subset"]["species_id"]]
}

# Haul UID reference dictionary
HAUL_UID_CONFIG = {
    "ship_id": {
        "US": config_params["biodata_subset"]["ship_us"], 
        "CAN": config_params["biodata_subset"]["ship_can"]
    },
    "survey_id": {
        "US": config_params["biodata_subset"]["survey_us"], 
        "CAN": config_params["biodata_subset"]["survey_can"]
    },
    "species_id": config_params["biodata_subset"]["species_id"],
    "haul_offset": config_params["biodata_subset"]["can_haul_offset"]
}

# Biodata ingestion parameters
RENAME_BIODATA_COLUMNS = config_params["rename_biodata_columns"]

# Biodata sex label mapping
BIODATA_SEX = {
    "sex": {
        1: "male",
        2: "female",
        3: "unsexed"
    }
}

# Ingest biodata
dict_df_bio = ingest.load_biological_data(
    biodata_filepath=BIODATA_FILE, 
    biodata_sheet_map=BIODATA_SHEETS, 
    column_name_map=FEAT_TO_ECHOPOP_BIODATA_COLUMNS, 
    survey_subset=BIODATA_SHIP_SPECIES, 
    biodata_label_map=BIODATA_SEX,
    haul_uid_config=HAUL_UID_CONFIG,
)

# Remove specimen-only hauls
feat_biology.drop_specimen_only_hauls(dict_df_bio)

YAML configuration

# Import packages
from pathlib import Path
from echopop import ingest
import yaml
from echopop.survey import biology

# Read in configuration
config_path = Path("configs/biodata_setup.yaml")
config_params = yaml.safe_load(config_path.read_text(encoding="utf-8"))

# Root data directory
DATA_ROOT = Path(config_params["data_root"])

# Biodata file [Excel]
BIODATA_FILE = DATA_ROOT / config_params["biodata_file"]

# Biodata file sheetnames
BIODATA_SHEETS = config_params["biodata_sheets"]

# Different biodata processing parameters required for parsing the Excel-formatted dataset
BIODATA_SHIP_SPECIES = {
    "ships": {
        config_params["biodata_subset"]["ship_us"]: {
            "country": "US",
            "survey": config_params["biodata_subset"]["survey_us"] 
        },
        config_params["biodata_subset"]["ship_can"]: {
            "country": "CAN",
            "survey": config_params["biodata_subset"]["survey_can"],
            "haul_offset": config_params["biodata_subset"]["can_haul_offset"]
        }
    },
    "species_code": [config_params["biodata_subset"]["species_id"]]
}

# Haul UID reference dictionary
HAUL_UID_CONFIG = {
    "ship_id": {
        "US": config_params["biodata_subset"]["ship_us"], 
        "CAN": config_params["biodata_subset"]["ship_can"]
    },
    "survey_id": {
        "US": config_params["biodata_subset"]["survey_us"], 
        "CAN": config_params["biodata_subset"]["survey_can"]
    },
    "species_id": config_params["biodata_subset"]["species_id"],
    "haul_offset": config_params["biodata_subset"]["can_haul_offset"]
}

# Biodata ingestion parameters
RENAME_BIODATA_COLUMNS = config_params["rename_biodata_columns"]

# Biodata sex label mapping
BIODATA_SEX = {
    "sex": {
        1: "male",
        2: "female",
        3: "unsexed"
    }
}

# Ingest biodata
dict_df_bio = ingest.load_biological_data(
    biodata_filepath=BIODATA_FILE, 
    biodata_sheet_map=BIODATA_SHEETS, 
    column_name_map=RENAME_BIODATA_COLUMNS, 
    survey_subset=BIODATA_SHIP_SPECIES, 
    biodata_label_map=BIODATA_SEX,
    haul_uid_config=HAUL_UID_CONFIG,
)

# Remove specimen-only hauls
feat_biology.drop_specimen_only_hauls(dict_df_bio)

This approach works well because users edit configuration values while the workflow logic remains stable. The most important safeguards are validating required keys before assignment and keeping key names consistent across projects so old configuration files do not silently break new workflow versions.

Conditional logic in configuration#

Configurations can be used to control the execution flow of a Python package by defining logic flags. By including specific key-value pairs—such as boolean toggles or categorical strings—the configuration file acts as a set of instructions that triggers specific if-else blocks within the code. This allows for the dynamic enabling or disabling of workflow steps without modifying the underlying script. For instance, a parameter called drop_specimen_hauls can be added to the configuration which controls whether the following line is executed:

feat_biology.drop_specimen_only_hauls(dict_df_bio)

JSON

{
  "data_root": "C:/Data/EchopopData",
  "biodata_file": "survey_biodata.xlsx",
  "biodata_sheets": {
    "catch": "biodata_catch",
    "length": "biodata_length",
    "specimen": "biodata_specimen"
  },
  "biodata_subset": {
    "species_id": 22500,
    "ship_us": 160,
    "ship_can": 584,
    "survey_us": 201906,
    "survey_can": 2019097,
    "can_haul_offset": 200,
  },  
  "rename_biodata_columns": {
    "frequency": "length_count",
    "haul": "haul_num",
    "haul ": "haul_num",
    "weight_in_haul": "weight"
  },
  "drop_specimen_hauls": true
}

YAML

data_root: C:/Data/EchopopData
biodata_file: survey_biodata.xlsx
biodata_sheets:
  catch: biodata_catch
  length: biodata_length
  specimen: biodata_specimen
biodata_subset:
  species_id: 22500
  ship_us: 160
  ship_can: 584
  survey_us: 201906
  survey_can: 2019097
  can_haul_offset: 200
rename_biodata_columns:
  frequency: length_count
  haul: haul_num
  "haul ": haul_num
  weight_in_haul: weight
drop_specimen_hauls: True

The loaded configuration can then be applied by inserting an if-else block into the code snippet.

Original

# Remove specimen-only hauls
feat_biology.drop_specimen_only_hauls(dict_df_bio)

Configured if-else check

# Remove specimen-only hauls only when configured
if config_params["drop_specimen_hauls"]:
    feat_biology.drop_specimen_only_hauls(dict_df_bio)

This if-else block makes the decision whether to drop the specimen-only hauls to be entirely configurable by the user without needing to modify the script itself every time.

Dynamic path selection#

Beyond simple parameter adjustments and if-else blocks, configuration flags can be utilized for dynamic path selection. This enables the user to instruct the software to execute a completely different branch of analysis. For instance, the workflow may be designed to trigger a specific biodata ingestion method such as biodata_format = "excel" for load_biological_data and biodata_format = "views" for load_biodata_views.

JSON

{
  "data_root": "C:/Data/EchopopData",
  "biodata_file": "survey_biodata.xlsx",
  "biodata_sheets": {
    "catch": "biodata_catch",
    "length": "biodata_length",
    "specimen": "biodata_specimen"
  },
  "biodata_subset": {
    "species_id": 22500,
    "ship_us": 160,
    "ship_can": 584,
    "survey_us": 201906,
    "survey_can": 2019097,
    "can_haul_offset": 200,
  },  
  "rename_biodata_columns": {
    "frequency": "length_count",
    "haul": "haul_num",
    "haul ": "haul_num",
    "weight_in_haul": "weight"
  },
  "biodata_format": "excel",
  "drop_specimen_hauls": true
}

YAML

data_root: C:/Data/EchopopData
biodata_file: survey_biodata.xlsx
biodata_sheets:
  catch: biodata_catch
  length: biodata_length
  specimen: biodata_specimen
biodata_subset:
  species_id: 22500
  ship_us: 160
  ship_can: 584
  survey_us: 201906
  survey_can: 2019097
  can_haul_offset: 200
rename_biodata_columns:
  frequency: length_count
  haul: haul_num
  "haul ": haul_num
  weight_in_haul: weight
biodata_format: excel
drop_specimen_hauls: True

This mechanism allows for a single workflow to support multiple, distinct operational modes. Users can toggle between these predefined branches to suit different data types or analysis objects without needing to modify the internal script structure. This allows the code snippet to be modified to:

Original

# Ingest biodata
dict_df_bio = ingest.load_biological_data(
    biodata_filepath=BIODATA_FILE, 
    biodata_sheet_map=BIODATA_SHEETS, 
    column_name_map=FEAT_TO_ECHOPOP_BIODATA_COLUMNS, 
    survey_subset=BIODATA_SHIP_SPECIES, 
    biodata_label_map=BIODATA_SEX,
    haul_uid_config=HAUL_UID_CONFIG,
)

# Remove specimen-only hauls
feat_biology.drop_specimen_only_hauls(dict_df_bio)

Biodata ingestion configuration

# Ingest biodata based on configured typing
if config_params["biodata_format"] == "excel":
    dict_df_bio = ingest.load_biological_data(
        biodata_filepath=BIODATA_FILE, 
        biodata_sheet_map=BIODATA_SHEETS, 
        column_name_map=RENAME_BIODATA_COLUMNS, 
        survey_subset=BIODATA_SHIP_SPECIES, 
        biodata_label_map=BIODATA_SEX,
        haul_uid_config=HAUL_UID_CONFIG,
    )
    # Remove specimen-only hauls only when configured
    # ---- ! This is ONLY compatible for when the Excel file format is ingested
    if config_params["drop_specimen_hauls"]:
        biology.drop_specimen_only_hauls(dict_df_bio)
elif config_params["biodata_format"] == "views":
    dict_df_bio = ingest.load_biodata_views(
        biodata_filepaths=BIODATA_FILE,
        column_name_map=RENAME_BIODATA_COLUMNS,
        survey_subset=BIODATA_SHIP_SPECIES, 
        biodata_label_map=BIODATA_SEX,
        haul_uid_config=HAUL_UID_CONFIG,
    )
else:
    raise KeyError(
        f"Configuration parameter 'biodata_format' must be either 'excel' or 'views'. "
        f"Got: '{config_params['biodata_format']}'!"
    )

Modular configuration management#

For more complicated workflows, modular configuration management separates settings by stage instead of storing everything in one large, cumbersome file. For instance, one configuration file could be assigned to data ingestion and another for stratification or analysis settings. This separation compartmentalizes stage-specific settings, reducing coupling between steps and lowering the risk that downstream edits unintentionally alter validated upstream parameters.

Common failure modes in non-modular setups include:

Shared keys across stages (namespace collision)
The same key name is intentionally reused in multiple workflow stages. Editing it for one stage also edits another stage.
Example: survey_year drives both ingestion file selection and stratification binning
Global configuration mutation (runtime side effect)
Code changes configuration values in-memory during execution, so later or repeated steps read modified values.
Example: a function sets config["species_id"] = override_ride and that persists.
Copy-paste drift (versioning/editing error)
Separate run files diverge inconsistently due to manual edits, creating mismatched parameter sets.
Example: ship_can is updated but survey_can is unchanged and left stale.
Implicit cross-stage dependency (hidden coupling)
A parameter appears stage-local but is also used elsewhere via undocumented logic.
Example: stratum_column is reused to build an ingestion join key.

Benefits of a modular approach

Isolation of parameters: Stage-specific settings stay in their own file, which reduces clutter and minimizes cross-stage edits.
Component reusability: One ingestion configuration can be paired with many stratification configurations for controlled comparisons.
Workflow scalability: New stages can be added by introducing a new configuration module without restructuring existing files.

Example layout#

One part of an example workflow may be broken up into these three stages:

biodata ingestion → stratification ingestion and apply stratification → bin age and length into their respective distributions

A configuration for each stage can then be created and maintained, such as within a directory like:

workflow_configs_hake2026/
  biodata_ingestion.yaml
  stratification.yaml
  biodata_analysis.yaml

Configuration file format

While this example uses YAML files, modular configuration files can also be JSON or a mix of multiple file formats.

../_images/modular_configuration_flow.svg

In this pattern, biodata_ingestion.yaml defines source files, sheet mappings, and other parameters as shown in previous configuration examples. stratification.yaml defines source files and other parameters for all stratification settings. And biodata_analysis.yaml stores information concerning the discretized age and length distributions.

biodata_ingestion.yaml

data_root: C:/Data/EchopopData
biodata_file: survey_biodata.xlsx
biodata_sheets:
  catch: biodata_catch
  length: biodata_length
  specimen: biodata_specimen
biodata_subset:
  species_id: 22500
  ship_us: 160
  ship_can: 584
  survey_us: 201906
  survey_can: 2019097
  can_haul_offset: 200
rename_biodata_columns:
  frequency: length_count
  haul: haul_num
  "haul ": haul_num
  weight_in_haul: weight
biodata_format: excel
drop_specimen_hauls: True

stratification.yaml

data_root: C:/Data/EchopopData
strata_file: survey_strata.xlsx
strata_sheets: 
    inpfc: INPFC
    ks: "Base KS"
rename_strata_columns:
  fraction_hake: nasc_proportion
  haul: haul_num
  stratum: stratum_num
default_stratum: 0
stratum_source: ks
stratum_name: stratum_ks
join_method: uid

biodata_analysis.yaml

age:
    start: 1
    end: 22
    count: 22
    column: age
length:
    start: 2
    end: 80
    count: 40
    column: length

The workflow then loads all three and applies them to the relevant function calls.

Show code cell source

Hide code cell source

# Import packages
from pathlib import Path
import yaml
import numpy as np

from echopop import ingest, utils
from echopop.workflows.nwfsc_feat import biology

# Read in configuration
config_ingest = yaml.safe_load(Path("configs/biodata_ingestion.yaml").read_text(encoding="utf-8"))
config_strata = yaml.safe_load(Path("configs/stratification.yaml").read_text(encoding="utf-8"))
config_analysis = yaml.safe_load(Path("configs/biodata_analysis.yaml").read_text(encoding="utf-8"))


# ---------------------------
# Stage 1: Biodata ingestion
# ---------------------------
# Root data directory
DATA_ROOT = Path(config_ingest["data_root"])

# Biodata file [Excel]
BIODATA_FILE = DATA_ROOT / config_ingest["biodata_file"]

# Biodata file sheetnames
BIODATA_SHEETS = config_ingest["biodata_sheets"]

# Different biodata processing parameters required for parsing the Excel-formatted dataset
BIODATA_SHIP_SPECIES = {
    "ships": {
        config_ingest["biodata_subset"]["ship_us"]: {
            "country": "US",
            "survey": config_ingest["biodata_subset"]["survey_us"] 
        },
        config_ingest["biodata_subset"]["ship_can"]: {
            "country": "CAN",
            "survey": config_ingest["biodata_subset"]["survey_can"],
            "haul_offset": config_ingest["biodata_subset"]["can_haul_offset"]
        }
    },
    "species_code": [config_ingest["biodata_subset"]["species_id"]]
}

# Haul UID reference dictionary
HAUL_UID_CONFIG = {
    "ship_id": {
        "US": config_ingest["biodata_subset"]["ship_us"], 
        "CAN": config_ingest["biodata_subset"]["ship_can"]
    },
    "survey_id": {
        "US": config_ingest["biodata_subset"]["survey_us"], 
        "CAN": config_ingest["biodata_subset"]["survey_can"]
    },
    "species_id": config_ingest["biodata_subset"]["species_id"],
    "haul_offset": config_ingest["biodata_subset"]["can_haul_offset"]
}

# Biodata ingestion parameters
RENAME_BIODATA_COLUMNS = config_ingest["rename_biodata_columns"]

# Biodata sex label mapping
BIODATA_SEX = {
    "sex": {
        1: "male",
        2: "female",
        3: "unsexed"
    }
}

# Ingest biodata based on configured typing
if config_ingest["biodata_format"] == "excel":
    dict_df_bio = ingestion.load_biological_data(
        biodata_filepath=BIODATA_FILE, 
        biodata_sheet_map=BIODATA_SHEETS, 
        column_name_map=FEAT_TO_ECHOPOP_BIODATA_COLUMNS, 
        survey_subset=BIODATA_SHIP_SPECIES, 
        biodata_label_map=BIODATA_SEX,
        haul_uid_config=HAUL_UID_CONFIG,
    )
    # Remove specimen-only hauls only when configured
    # ---- ! This is ONLY compatible for when the Excel file format is ingested
    if config_ingest["drop_specimen_hauls"]:
        feat_biology.drop_specimen_only_hauls(dict_df_bio)
elif config_ingest["biodata_format"] == "views":
    dict_df_bio = ingestion.load_biodata_views(
        biodata_filepaths=BIODATA_FILE,
        column_name_map=FEAT_TO_ECHOPOP_BIODATA_COLUMNS,
        survey_subset=BIODATA_SHIP_SPECIES, 
        biodata_label_map=BIODATA_SEX,
        haul_uid_config=HAUL_UID_CONFIG,
    )
else:
    raise KeyError(
        f"Configuration parameter 'biodata_format' must be either 'excel' or 'views'. "
        f"Got: '{config_ingest['biodata_format']}'!"
    )

# ------------------------------------
# Stage 2: Stratification + join stage
# ------------------------------------
# Load strata file
DATA_STRATA_ROOT = Path(config_strata["data_root"])
df_dict_strata = ingestion.load_strata(
    strata_filepath=DATA_STRATA_ROOT / config_strata["strata_file"], 
    strata_sheet_map=config_strata["strata_sheets"], 
    column_name_map=config_strata["rename_strata_columns"],
    haul_uid_config=HAUL_UID_CONFIG,
)

# Apply strata to biodata
if config_strata["join_method"] == "haul":
    dict_df_bio = ingest.join_strata_by_haul(
        data=dict_df_bio,
        strata=df_dict_strata[config_strata["stratum_source"]],
        default_stratum=config_strata["default_stratum"],
        stratum_name=config_strata["stratum_name"]
    )
elif config_strata["join_method"] == "uid":
    dict_df_bio = ingest.join_strata_by_uid(
        data=dict_df_bio,
        strata=df_dict_strata[config_strata["stratum_source"]],
        default_stratum=config_strata["default_stratum"],
        stratum_name=config_strata["stratum_name"]
    )
else:
    raise KeyError(
        f"Configuration parameter 'join_method' must be either 'haul' or 'uid'. "
        f"Got: '{config_strata['biodata_format']}'!"
    )    

# ---------------------------
# Stage 3: Analysis bin setup
# ---------------------------
# Binify data
# --- Age-bins
AGE_BINS = np.linspace(
    start=config_analysis["age"]["start"], 
    stop=config_analysis["age"]["end"], 
    num=config_analysis["age"]["count"]
)
utils.binify(
    data=dict_df_bio, bins=AGE_BINS, bin_column=config_analysis["age"]["column"],
)

# LENGTH-BINS
LENGTH_BINS = np.linspace(
    start=config_analysis["length"]["start"], 
    stop=config_analysis["length"]["end"], 
    num=config_analysis["length"]["count"]
)
utils.binify(
    data=dict_df_bio, bins=LENGTH_BINS, bin_column=config_analysis["length"]["column"], 
)

Configure workflow scripts

Contents

Configure workflow scripts#

Configuration formats#

JSON#

YAML#

Modifying workflows#

Loading JSON and YAML into this workflow block#

Applying user configurations#

Conditional logic in configuration#

Dynamic path selection#

Modular configuration management#

Example layout#