Run year-specific workflows

This notebook presents an example implementation of multi-year orchestration for FEAT year scripts, grounded directly in the year_specific/ source files. The discussion keeps the same technical framing used across this documentation section: workflow-stage composition follows the assumptions in the workflow overview, orchestration behavior follows the execution model in the orchestration guide, and CLI argument flow follows the hook patterns established in the CLI walkthrough.

Architecture overview

The orchestration model uses a single dispatcher to route execution to a selected year script (hake_1995.py through hake_2023.py), while shared CLI hooks expose runtime controls such as verbosity and report-comparison behavior.

In practice, the flow starts from one of three surfaces: direct CLI invocation, wrapper-script execution, or notebook orchestration. After the entrypoint call reaches year_specific_workflow.py, the dispatcher resolves the year target, forwards passthrough arguments, and runs the selected script through a consistent ingestion-to-reporting sequence. Shared utilities in cli_utils.py provide reusable hook behavior inside each year script, which keeps orchestration logic centralized without flattening year-specific parameter definitions.

Source files and responsibilities

The orchestration contract is implemented through three cooperating source layers. The dispatcher layer in year_specific_workflow.py parses --year, --verbose, and --compare, routes to hake_<year>.py using runpy.run_path, and preserves passthrough arguments through parse_known_args(). The shared-hook layer in cli_utils.py defines get_verbose() and get_compare(), which year scripts consume to adapt runtime behavior based on CLI state. The year-script layer (hake_1995.py through hake_2023.py) contains year-specific file mappings and parameters while still following the same FEAT stage pattern, which allows local specialization without fragmenting the orchestration interface.

Dispatcher CLI arguments

The dispatcher interface is intentionally small, and each argument has a distinct role:

• --year (required): Selects which year script to execute (for example, hake_2019 or hake_2023). The dispatcher resolves this into hake_<year>.py and runs that file.

• --verbose (optional flag): Enables more detailed stage-level logging output while the selected year script runs.

• --compare (optional flag): Enables the EchoPro-vs-Echopop comparison pathway in year scripts that support comparison outputs.

Because the dispatcher uses parse_known_args(), additional downstream arguments can still be passed through to the selected year script without being dropped.

Workflow parameters across year-specific scripts

Parameter blocks in year_specific/hake_*.py follow a shared schema even when year-specific files diverge on source files or survey composition.

Parameter blocks

In this context, a parameter block is the contiguous top-level configuration section in each hake_<year>.py file that appears before the START OF PROCESSING SCRIPT marker. It centralizes runtime controls (paths, toggles, survey IDs, and model input file mappings) so the core processing flow can stay structurally consistent across years. These parameter blocks would likely benefit from incorporating external year-specific configuration files so the underlying workflow scripts do not have to be modified.

A simplified example adapted from hake_2019.py:

# PARAMETER ENTRY (example shape)
try:
    VERBOSE = cli_utils.get_verbose()
except Exception:
    VERBOSE = True

DATA_ROOT = Path("C:/Data/EchopopData/echopop_2019")
REPORTS_DIR = DATA_ROOT / "reports_updated_biodata"

try:
    COMPARE = cli_utils.get_compare()
    ECHOPRO_REPORTS_DIR = DATA_ROOT / "reports_echopro"
    COMPARISONS_DIR = DATA_ROOT / "comparisons_updated_biodata"
except Exception:
    COMPARE = False

NASC_PREPROCESSED = True
NASC_EXPORTS_FILES = DATA_ROOT / "Exports/US&CAN_detailsa_2019_table2y+_ALL_final - updated.xlsx"
NASC_EXPORTS_SHEET = "Sheet1"

SPECIES_ID = 22500
SHIP_US = 160
SURVEY_US = 201906

KRIGING_MESH_FILE = DATA_ROOT / "Kriging_files/.../krig_grid2_5nm_cut_centroids_2013.xlsx"
KRIGING_MESH_SHEET = "krigedgrid2_5nm_forChu"

Simplified means simplified

The above snippet does not represent the entirety of the hake_2019.py parameter block.

The inventory below reflects variables defined before the START OF PROCESSING SCRIPT marker across all available year scripts (hake_1995.py, …, hake_2023.py). This cross-year pattern keeps dispatcher behavior stable while allowing each year script to refine ingestion and survey harmonization details without changing the external orchestration interface.

Parameters

The tables below indicate the various parameters/variables used throughout the year-specific workflow scripts.

Workflow

These are boolean parameters that instruct the workflow on how to ingest and process survey data.

Variable(s)	Functional role	Year coverage
VERBOSE	Enables stage-level logger output (logging.INFO) when the verbose hook is active.	All years
COMPARE, SHOW_PLOT	Enables and configures EchoPro-vs-Echopop comparison branch outputs.	All years
NASC_PREPROCESSED	Switches between preformatted NASC input and raw-export processing pathways.	2003, 2005, 2007, 2009, 2011, 2012, 2013, 2015, 2017, 2019, 2021, 2023
OPTIMIZE_VARIOGRAM	Toggles variogram optimization versus fixed-parameter operation.	All years
REMOVE_AGE1, AGE1_DOMINATED_HAULS	Controls age-1 exclusion and haul-level override behavior for age composition handling.	All years

Filepath

These parameters define the directories and filepaths of input datasets, generated reports, and visualizations.

Variable(s)	Functional role	Year coverage
BIODATA_FILE, BIODATA_SHEETS	Defines biological input file(s) and sheetname map.	All listed years
DATA_ROOT, REPORTS_DIR	Root pathing for input assets and report/export destinations.	All listed years
ECHOPRO_REPORTS_DIR, COMPARISONS_DIR	Directory where EchoPro reports are located and where to store Echopop-vs-EchoPro visualizations.	All listed years
GEOSTRATA_FILE, GEOSTRATA_SHEETS	Geographic stratification source and sheet mapping (ks, inpfc).	All listed years
HAUL_STRATA_FILE, HAUL_STRATA_SHEETS	Haul-based stratification source and sheet mapping (ks, inpfc).	All listed years
ISOBATH_FILE, ISOBATH_SHEET	Isobath coordinate-transform source used in geostatistical stages.	All listed years
KRIGING_MESH_FILE, KRIGING_MESH_SHEET	Spatial mesh source used by kriging/grid conversion stages.	All listed years
KRIGING_VARIOGRAM_PARAMETERS_FILE, KRIGING_VARIGORAM_PARAMETERS_SHEET	Variogram/kriging settings source and worksheet key.	All listed years
NASC_EXPORTS_FILES, NASC_EXPORTS_SHEET	NASC input workbook path and worksheet target used by ingestion.	All listed years
TRANSECT_BOUNDARY_FILE, TRANSECT_BOUNDARY_SHEET	Transect-boundary lookup for transect/regional assignment steps.	1995, 1998, 2001, 2003, 2005, 2007, 2009, 2011, 2012, 2013
TRANSECT_REGION_HAUL_FILE, TRANSECT_REGION_HAUL_SHEET	Region/haul mapping inputs used in mid-period transect regionalization.	TRANSECT_REGION_HAUL_FILE: 2003, 2005, 2007, 2009, 2011, 2012, 2013; TRANSECT_REGION_HAUL_SHEET: same set excluding 2011

Biodata

These parameters are required for ingesting and processing the biological data.

Variable(s)	Functional role	Year coverage
BIODATA_SHIP_SPECIES	Defines biological input file(s), sheet map, and ship/survey/species parsing contract.	All listed years
SPECIES_ID, SHIP_US, SURVEY_US	Core U.S.-side species and survey identifiers used by UID construction and filtering.	SPECIES_ID: all years; SHIP_US/SURVEY_US: all years except 2003
SHIP_CAN, SURVEY_CAN, CAN_HAUL_OFFSET	Canadian survey IDs and haul-offset handling for joint-survey harmonization.	SHIP_CAN/SURVEY_CAN: 1998, 2001, 2003, 2009, 2011, 2012, 2013, 2015, 2017, 2019, 2021, 2023; CAN_HAUL_OFFSET: same set excluding 2003

Filtering

These parameters are used by the workflow for filtering different datasets.

Variable(s)	Functional role	Year coverage
AGE1_DOMINATED_HAULS	Hauls dominated by age-1 fish that are removed from the acoustic and biological datasets.	2011
SURVEY_FILTER	Applies survey-ID filtering logic in mixed-source ingestion windows.	1995, 1998, 2001, 2003, 2005, 2007, 2009, 2011, 2012, 2013

Additional

Throughout the workflow there are additional parameters/variables that are defined that may or may not be year-specific. It is important that the entire workflow script is inspected to avoid copy-paste drift.

Variable(s)	Functional role
BIODATA_SEX	Dictionary map for replacing numeric sex identifiers with explicit male, female, and unsexed labels.
CLASS_REGIONS	A list of class region names used for filtering the raw NASC dataset.
ECHOPRO_TO_ECHOPOP_FILE_MAP	Dictionary map that links EchoPro and Echopop reports of the same type.
FEAT_STRATEGY_KWARGS	Dictionary containing keyword arguments specific to the western-boundary-based adaptive search strategy used for kriging.
FEAT_TO_ECHOPOP_COLUMNS	Dictionary map for renaming ingested NASC data columns.
FEAT_TO_ECHOPOP_BIODATA_COLUMNS	Dictionary map for renaming ingested biological data columns.
FEAT_TO_ECHOPOP_GEOSTATS_PARAMS_COLUMNS	Dictionary map for renaming ingested kriging and variogram model parameter names.
FEAT_TO_ECHOPOP_GEOSTRATA_COLUMNS	Dictionary map for renaming ingested latitude-based stratification data columns.
FEAT_TO_ECHOPOP_STRATA_COLUMNS	Dictionary map for renaming ingested length-based stratification data columns.
JOLLYHAMPTON_PARAMETERS	Dictionary containing parameters for stratified random sampling analysis.
KRIGING_PARAMETERS	Dictionary containing kriging parameters required for interpolation.
TRANSECT_REGION_FILE_RENAME	Dictionary map for renaming ingested transect-region-haul key columns.
SURVEY_FILTER	Applies survey-ID filtering logic in mixed-source ingestion windows.
VARIOGRAM_PARAMETERS	Dictionary containing variogram parameters required for ordinary kriging.

Dispatcher CLI design

The dispatcher year_specific_workflow.py is intentionally thin: it resolves a year target, forwards runtime arguments, and delegates execution without embedding survey-domain logic. This separation keeps routing behavior stable while allowing year scripts to evolve independently.

year_specific_workflow.py

# CLI hook
import argparse
import runpy
import sys
import os

# Set up argument parser and parsing
parser = argparse.ArgumentParser()
parser.add_argument("--year", required=True, help="Which workflow script to run (e.g. hake_1995)")
parser.add_argument("--verbose", action="store_true", help="Enable verbose logging")
parser.add_argument("--compare", action="store_false", help="Compare output reports to EchoPro")
args, unknown = parser.parse_known_args()

# Pass additional CLI args to the target script
sys.argv = [args.year + ".py"] + unknown
if args.verbose:
    sys.argv.append("--verbose")
if args.compare:
    sys.argv.append("--compare")

# Get the script path
script_path = os.path.join(os.path.dirname(__file__), f"{args.year}.py")
runpy.run_path(script_path, run_name="__main__")

Two implementation details matter for orchestration reliability: parse_known_args() preserves flags intended for downstream scripts, and runpy.run_path(...) executes the selected year module in-process without requiring duplicated launcher code per survey year.

Shared hooks and workflow integration

Year-specific scripts use shared utility hooks from cli_utils.py to read CLI state and control runtime behavior, especially logging and EchoPro-Echopop comparison pathways. Centralizing these checks in one utility module reduces drift across year-specific files and keeps hook semantics consistent.

cli_utils.py

import argparse

def get_verbose():
    parser = argparse.ArgumentParser()
    parser.add_argument("--verbose", action="store_true", help="Enable verbose logging")
    args, unknown = parser.parse_known_args()
    return args.verbose

def get_compare():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--compare", action="store_false", help="Compare generated reports to EchoPro"
    )
    args, unknown = parser.parse_known_args()
    return args.compare

This gets incorporated into each workflow file by adding this code snippet:

# Import 
from echopop.workflow_examples import cli_utils

# Print console logging messages
# ---- When set to `True`, logging information will be printed in the terminal/console as the 
# ---- script progresses
try: 
    # ---- FOR CLI USE
    VERBOSE = cli_utils.get_verbose()
except Exception:
    # ---- FOR INTERACTIVE REPL USE
    VERBOSE = True

# Compare to EchoPro reports?
try:
    # ---- FOR CLI USE
    COMPARE = cli_utils.get_compare()
    ECHOPRO_REPORTS_DIR = DATA_ROOT / "reports_echopro"
    COMPARISONS_DIR = DATA_ROOT / "comparisons_updated_biodata"
    SHOW_PLOT = False
except Exception:
    # ---- FOR INTERACTIVE REPL USE
    COMPARE = False

The try/except fallback pattern allows the same year script to run cleanly from CLI, wrapper, and interactive contexts, while still defaulting to predictable behavior when flags are absent.

CLI usage examples by platform

In each platform variant below, --year selects the script target and optional flags tune runtime behavior without changing year-file source code. The commands are intentionally parallel so users can move between shells without learning a different orchestration contract.

Windows CMD

python year_specific_workflow.py --year hake_2023
python year_specific_workflow.py --year hake_2021 --verbose
python year_specific_workflow.py --year hake_2019 --verbose --compare

Windows PowerShell

python year_specific_workflow.py --year hake_2023
python year_specific_workflow.py --year hake_2021 --verbose
python year_specific_workflow.py --year hake_2019 --verbose --compare

Linux/macOS bash

python3 year_specific_workflow.py --year hake_2023
python3 year_specific_workflow.py --year hake_2021 --verbose
python3 year_specific_workflow.py --year hake_2019 --verbose --compare

Notebook-driven orchestration

A notebook can serve as a lightweight orchestration surface when live logs, incremental diagnostics, and reproducible command history are needed in one place. The pattern below launches the same dispatcher command used in terminal workflows and streams output in real time, so notebook execution preserves operational visibility rather than hiding stage progress until process completion. The co

A notebook serves as a lightweight orchestration surface when live logs, incremental diagnostics, and reproducible command history are needed in one place. This pattern links interactive data exploration and CLI execution. By wrapping a dispatcher command (i.e., year_specific_workflow.py) the notebook ensures that stage progress, warnings, and verbose telemetry are captured and displayed as they occur if --verbose is flagged.

Shell escape (!)

The shell escape (!) is the most direct and simplest method. It is useful for one-off commands where logging information, stage progress, etc., are not as important to users. The command can be simply added to a Python code cell:

# Set year
year = 1995

# Execute
!python year_specific_workflow.py --year hake_{year} --verbose

Programmatic dispatch

The subprocess module allows for logging information to be prase while it is running. There are additional benefits to using subprocess, including reliable termination when the notebook kernel is interrupted and simplified debugging by ensuring errors are visible exactly when they happen.

import subprocess

# Parameterize the execution
year = 2023
cmd = ["python", "year_specific_workflow.py", "--year", f"hake_{year}", "--verbose"]

print(f"Starting workflow for {year}...\n")

# Launch the process with piped streams
process = subprocess.Popen(
    cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,  # Redirect errors to the standard output stream
    text=True,                 # Decode bytes to strings automatically
    bufsize=1                  # Line-buffered for immediate printing
)

# Iterate over the output stream until the process terminates
try:
    for line in process.stdout:
        print(line, end="")
except KeyboardInterrupt:
    process.kill()
    print("\n[Workflow interrupted by user; process terminated]")

process.wait()
if process.returncode == 0:
    print(f"\nWorkflow for {year} completed successfully.")
else:
    print(f"\nWorkflow failed with exit code {process.returncode}.")

When --verbose and ran with subprocess, the following logs are printing in the terminal/console:

Reading pre-generated NASC export file: 'C:/Data/EchopopData/echopop_1995/Exports/US&CAN_detailsa_1995_table1y+_ALL_final.xlsx'.
---- Converting AFSC NASC export format to FEAT
     Default interval distance: 0.5 nmi
     Default transect spacing: 10.0 nmi
     Including transects: 1 to 400
---- Filtering out off-effort transect intervals based on: C:\Data\EchopopData\echopop_1995\Kriging_files\Kriging_grid_files\Transect Bounds to 2011.xlsx
     Survey filter: 'survey == 199510'
!!! [1995] WARNING:
3925 rows had missing vessel log distances.
These are incompatible with the workflow and will therefore be removed accordingly.
!!! [1995] WARNING:
Longitude values in 'df_nasc' are in deg.W. However, they must be in deg.E. These have been converted accordingly.
!!! [1995] WARNING:
Columns 'region_id', 'layer_height', and 'layer_mean_depth' missing. Stand-in values have been added.
Beginning biodata ingestion for: 'C:/Data/EchopopData/echopop_1995/Biological/1995-2023_biodata_redo.xlsx'.
Loading stratification files...
Load in haul-based stratification: 'C:/Data/EchopopData/echopop_1995/Stratification/US&CAN strata 1995_final.xlsx'.
Dropping column 'length' from the haul-based stratification datasets.
Haul-based stratification loading complete
'df_dict_strata' created.
Load in geographic-based stratification: 'C:/Data/EchopopData/echopop_1995/Stratification/Stratification_geographic_Lat_1995.xlsx'.
Loading kriging mesh file: 'C:/Data/EchopopData/echopop_1995/Kriging_files/Kriging_grid_files/krig_grid2_5nm_cut_centroids_2013.xlsx'.
Loading isobath file: 'C:\Data\EchopopData\echopop_1995\Kriging_files\Kriging_grid_files\transformation_isobath_coordinates.xlsx'.
Loading variogram and kriging parameters: 'C:/Data/EchopopData/echopop_1995/Kriging_files/default_vario_krig_settings_orig.xlsx'.
Applying strata to datasets...
Applying haul-based strata to 'dict_df_bio' and 'df_nasc'.
     Default stratum: 0
     New columns:
         INPFC: 'stratum_inpfc' [2001: BIODATA ONLY]
         KS: 'stratum_ks' [2001: BIODATA ONLY]
Applying geographic-based strata to 'df_nasc' and 'df_mesh.
     New columns:
         INPFC: 'geostratum_inpfc'
         KS: 'geostratum_ks'
Strata application complete!
Binning biodata ['dict_df_bio'] into discrete age and length bins
     Age bins: [1, 2, 3, ..., 20, 21, 22]
     Length bins: [2.0, 4.0, 6.0, ... 76.0, 78.0, 80.0]
     New columns:
         Age: 'age_bin'
         Length: 'length_bin'
Age and length binning complete!
Fitting length-weight regression for each sex and for all fish.
C:\Users\Brandyn\GitHub\echopop\echopop\survey\biology.py:40: RankWarning: Polyfit may be poorly conditioned
  np.polyfit(np.log10(clean_data["length"]), np.log10(clean_data["weight"]), 1),
Fitting length-weight regression for each sex and for all fish complete!
Computing the mean weight per length bin for each sex and for all fish.
     Impute missing length bins using modeled weights: True     Minimum specimen count per bin: 5
Computing the counts per age- and length-bins across sex.
     Stratifying by: 'stratum_ks'     Grouping by: 'sex'
Computing number proportions across age and length bins
     Stratifying by: 'stratum_ks'
     Excluding: 'sex'='unsexed' from 'dict_df_counts['aged']'
Computing the summed weights per age- and length-bins across sex.
     Stratifying by: 'stratum_ks'     Grouping by: 'sex'     Excluding: 'sex'='unsexed'
Unaged binned weights require additional processing steps.
     Interpolating binned length-weight regression estimates: True
Computing weight proportions across age and length bins
     Stratifying by: 'stratum_ks'     Grouping by: 'sex'
Computing aged weight proportions...
Scaling unaged binned weights...
Computing unaged weight proportions
     Scaling weight proportions in reference to the aged estimates
Beginning inversion based on hake-specific TS-length regression coefficients
     Model: 20.0 x log[10](L) + -68.0
     Stratifying by: 'stratum_ks'
     Imputing missing strata: True
     Treating hauls as replicates: True
Inversion-class object 'invert_hake' created...
Number density inversion complete
     New column in 'df_nasc':
         Number density (animals nm^-2): 'number_density'
Converting number density estimates into biomass estimates
Defining transect interval distances...
     Along-transect interval distance (nmi) threshold: 0.5 nmi
Defining transect interval areas...
Compute interval abundances...
     Stratifying by: 'stratum_ks'
     Grouping by: 'sex'
     Excluding: 'sex'='unsexed' from 'dict_df_number_proportions'
Compute interval biomass...
     Stratifying by: 'stratum_ks'
     Grouping by: 'sex'

NASC to biomass conversion complete
     New columns in 'df_nasc':
         Sex-specific number densities (animals nmi^-2): 'number_density_female'/'number_density_male'
         Abundance (animals): 'abundance'/'abundance_female'/'abundance_male'
         Biomass density (kg nmi^-2): 'biomass_density'/'biomass_density_female'/'biomass_density_male'
         Biomass (kg): 'biomass'/'biomass_female'/'biomass_male'
Distribute population estimates across age- and length-bins
     Stratifying by: 'stratum_ks'
     Grouping by: 'sex'
Distributing abundances...
Distributing biomass...
Biomass distribution complete
'dict_transect_biomass_table' created.
Distributing biomass...
     Aged-only weight proportions: True
Beginning geostatistical analysis...
Transform spatial coordinates for 'df_nasc' and 'df_mesh'
     Reference coordinates: 'df_isobath'
     Longitudinal offset: -124.78338 deg.E
     Latitudinal offset: 45.0 deg.N
Coordinate transformation complete
     New columns:
          Transformed longitude: 'x'
          Transformed latitude: 'y'

Beginning variogram analysis
     Normalized lag resolution: 0.002
     Number of lags: 30

Computing the empirical variogram
     Variable: 'biomass_density'
     Applying azimuth angle filter: True
     Azimuth angle filter: 180.0 deg.

Optimizing variogram parameters using non-linear least-squares
     Model: Exponential-Bessel (['exponential', 'bessel'])
     Initial values:
          Nugget: 0.0
          Sill: 0.92
          Correlation range: 0.007
          Hole effect range: 0.0
          Decay power exponent: 1.0
Optimization arguments:
{'max_nfev': None, 'ftol': 1e-08, 'gtol': 1e-08, 'xtol': 1e-08, 'diff_step': None, 'tr_solver': 'exact', 'x_scale': 1.0, 'jac': '2-point'}
Variogram parameter fitting complete
     Best-fit parameters:
     {'nugget': 9.823973946277347e-24, 'sill': 0.9067943245742746, 'correlation_range': 0.009325484840718806, 'hole_effect_range': 5.723331256124487e-10, 'decay_power': 1.6893193068269476}
Beginning kriging analysis
     Using best-fit variogram model and parameters: True
     Kriging parameters:
          Normalized search radius: 0.027976454522156417
          Minimum nearest-neighbor count: 3
          Maximum nearest-neighbor count: 10
          Anisotropic aspect ratio: 0.001
Interpolating population estimates using ordinary kriging
     Variable: 'biomass_density'
     Extrapolation (full mesh): True
     Default mesh cell area: 6.25 nmi^2

C:\Users\Brandyn\GitHub\echopop\echopop\geostatistics\kriging.py:1414: UserWarning: 366 invalid kriged estimates (< 0.0) found. These have been replaced with '0.0'; however, this truncation may distort error distributions. If this is problematic, adjust the choice of variogram and kriging algorithm parameterization.
  kriged_results, _ = project_kriging_results(
Converting biomass density estimates into NASC
     Stratifying by: 'geostratum_ks'/'stratum_ks'
     Grouping by: 'sex'
     Using stratum weights for all fish: True
New column in 'df_kriged_results': 'biomass'
Biomass density to NASC conversion complete
    New columns in `df_kriged_results`
        Biomass (kg): 'biomass'/'biomass_female'/'biomass_male'
        Abundance (animals): 'abundance'/'abundance_female'/'abundance_male'
        NASC (m^2 nmi^-2): 'nasc'
Distribute kriged population estimates across age- and length-bins
     Stratifying by: 'stratum_ks'
     Grouping by: 'sex'
Distributing abundances...
Abundance distributions complete
'dict_kriged_abundance_table' created.
Scaling unaged abundance...
     Reference: Aged abundances
     Imputing missing bins: False
Distributing biomass...
Biomass distribution complete
'dict_kriged_biomass_table' created.
Scaling unaged biomass...
     Reference: Aged biomass
     Imputing missing bins: True
Consolidating abundance tables...
Abundance table complete
'df_kriged_abundance_table' created.
Consolidating biomass tables...
Beginning stratified analysis to estimate uncertainties (Jolly and Hampton, 1990)...
Running Jolly and Hampton (1990) algorithm for transect data
     Variable: 'biomass'
     Number of bootstrap replicates: 1000
     Stratum transect sampling proportion: 0.75
     Stratifying by: 'geostratum_ks'
Summarizing results....
     Confindence interval percentile: 0.95
     Confidence interval method: Jackknife studentized interval ('t-jackknife')
Creating virtual transects for kriged mesh data
     Stratifying by: 'geostratum_inpfc'
     Virtual transects per latitude: 5
Running Jolly and Hampton (1990) algorithm for kriged data
     Variable: 'biomass'
     Number of bootstrap replicates: 1000
     Virtual transect sampling proportion: 0.75
     Stratifying by: 'geostratum_ks'
Summarizing results....
     Confindence interval percentile: 0.95
     Confidence interval method: Jackknife studentized interval ('t-jackknife')
Writing reports to: 'C:/Data/EchopopData/echopop_1995/reports'.
Aged-length haul counts report saved to 'C:/Data/EchopopData/echopop_1995/reports/aged_length_haul_counts.xlsx'.
Total haul length counts report saved to 'C:/Data/EchopopData/echopop_1995/reports/total_length_haul_counts.xlsx'.
Kriged aged biomass mesh report saved to 'C:/Data/EchopopData/echopop_1995/reports/kriged_aged_biomass_mesh_full.xlsx'.
Kriged aged biomass mesh report saved to 'C:/Data/EchopopData/echopop_1995/reports/kriged_aged_biomass_mesh_nonzero.xlsx'.
Kriged mesh report saved to 'C:/Data/EchopopData/echopop_1995/reports/kriged_biomass_mesh_full.xlsx'.
Kriged mesh report saved to 'C:/Data/EchopopData/echopop_1995/reports/kriged_biomass_mesh_nonzero.xlsx'.
Kriged age-length abundance report saved to 'C:/Data/EchopopData/echopop_1995/reports/kriged_length_age_abundance_report.xlsx'.
Kriged age-length biomass report saved to 'C:/Data/EchopopData/echopop_1995/reports/kriged_length_age_biomass_report.xlsx'.
Kriging input report saved to 'C:/Data/EchopopData/echopop_1995/reports/kriging_input_report.xlsx'.
Transect age-length abundance report saved to 'C:/Data/EchopopData/echopop_1995/reports/transect_length_age_abundance_report.xlsx'.
Transect age-length biomass report saved to 'C:/Data/EchopopData/echopop_1995/reports/transect_length_age_biomass_report.xlsx'.
Transect aged biomass report saved to 'C:/Data/EchopopData/echopop_1995/reports/transect_aged_biomass_report_full.xlsx'.
Transect aged biomass report saved to 'C:/Data/EchopopData/echopop_1995/reports/transect_aged_biomass_report_nonzero.xlsx'.
Transect population results report saved to 'C:/Data/EchopopData/echopop_1995/reports/transect_population_results_full.xlsx'.
Transect population results report saved to 'C:/Data/EchopopData/echopop_1995/reports/transect_population_results_nonzero.xlsx'.

Batch orchestration across multiple years

For repeated execution across survey years, wrapper scripts provide a practical control layer that reuses the same dispatcher contract instead of duplicating orchestration logic in every environment. These scripts automate the loop through historical data while keeping failure handling explicit. This ensures that a multi-year run can be audited quickly, identifying exactly which years suceeded and which require further inspection.

Windows .bat

The Windows Batch format is compatible with the classic Command Prompt (cmd.exe). It utilizes the %errorlevel% variable to detect if the previous command succeeded or failed.

To run: Open Command Prompt, navigate to the folder, and type run_hake.bat.

@echo off
setlocal enabledelayedexpansion

set YEARS=1995 1998 2001 2003

for %%Y in (%YEARS%) do (
    echo [INFO] Running hake_%%Y ...
    python year_specific_workflow.py --year hake_%%Y --verbose

    if errorlevel 1 (
        echo ❌ Error encountered for year %%Y
    ) else (
        echo ✅ Completed year %%Y
    )
)

Windows PowerShell (.ps1)

PowerShell provides a more descriptive environment for modern Windows systems. It tracks the success of external applications via the $LASTEXITCODE variable.

To run: Open PowerShell, navigate to the folder, and type .\run_hake.ps1.

$years = @(1995, 1998, 2001, 2003)
$failedYears = @()

foreach ($year in $years) {
    Write-Host "Running workflow for year $year..."
    python year_specific_workflow.py --year "hake_$year" --verbose

    if ($LASTEXITCODE -eq 0) {
        Write-Host "✅ Completed year $year" -ForegroundColor Green
    } else {
        Write-Host "❌ Error encountered for year $year" -ForegroundColor Red
        $failedYears += $year
    }
}

if ($failedYears.Count -gt 0) {
    Write-Host "`nSummary of Failures: $($failedYears -join ', ')"
}

Linux/macOS bash (.sh)

Unix-based systems utilize Bash scripts. These scripts evaluate the exit status of the Python command directly within an if statement logic.

To run: Open Terminal, navigate to the folder, and type ./run_hake.sh.

#!/usr/bin/env bash
set -u

years=(1995 1998 2001 2003)
failed_years=()

for year in "${years[@]}"; do
  echo "Running workflow for year ${year}..."
  
  if python3 year_specific_workflow.py --year "hake_${year}" --verbose; then
    echo "✅ Completed year ${year}"
  else
    echo "❌ Error encountered for year ${year}"
    failed_years+=("$year")
  fi
done

if [ ${#failed_years[@]} -gt 0 ]; then
  echo -e "\nSummary of Failures: ${failed_years[*]}"
  exit 1
fi

Practical notes on CLI flags and logging

The interface between the orchestration layer and the underlying Python scripts relies on a standardized set of command-line flags. This consistency ensures that whether a task is launched from a notebook, a batch file, or a terminal, the behavior and output remain predictable.

Argument routing and configuration

Within this implementation, the --year flag serves as the primary routing key. It dictates which specific year-based data and logic the dispatcher executes. This approach allows a single entry point to handle a diverse array of survey years by dynamically loading the appropriate configuration.

The --verbose flag functions as a toggle for operational transparency. It controls whether detailed stage progress is emitted to the console. When scripts are configured to honor this flag, they provide a granular view of the internal workflow. The comparison pathway is handled through a shared “compare hook” used by both the dispatcher and the individual year scripts. This shared architecture ensures that data validation remains consistent across all execution environments.

Maintaining execution visibility

While the --verbose flag determines the volume of information, the orchestration layer must always be prepared to handle output as a continuous live feed. Year-specific workflows are designed to emit stage-level progress messages (using logging.info) only when explicitly requested. Maintaining an open communication channel between the process and the user is vital for the following reasons:

• Real-time transparency: When --verbose is included, there is an expectation of progression from the process. If the orchestration layer waits until the end to show the logs, users see nothing for the duration of the run, which often leads to the false assumption that the software has crashed.

• Immediate error report: Even in silent mode when --verbose is not flagged, failures and systme tracebacks are sent to the error stream. Handling this output in real time ensures these messages appear immediately. THis prevents the user from waiting through a long process that actually failed in the first few seconds.

By maintaining a direct line to the script output, the orchestration surface (whether a Jupyter Notebook or a Batch file) respects the user’s intent by providing total silence when preferred (no --verbose) or total transparency when needed (--verbose).