Age-1 exclusion workflow

Age-1 exclusion workflow#

Not reinventing the wheel

This notebook documents how to integrate age-1 exclusion into the previous year-specific workflow example. Rather than reproducing the full script, this page isolates the exact parameter triggers, conditional processing blocks, and output variables that enable the age-1 exclusion pathway. It also explains how to expose the exclusion behavior as a CLI flag so the same year-specific script can produce either an all-ages run or an age-2+ run without modifying the source code. The discussion uses the same technical framing as the workflow overview and builds directly on the year-specific workflows and CLI walkthrough pages.

Architecture overview#

The age-1 exclusion branch does not require a separate workflow script. It is implemented as a set of conditional blocks inside an existing year-specific script (hake_<year>.py), gated by two parameters: REMOVE_AGE1 and AGE1_DOMINATED_HAULS. The dispatcher (year_specific_workflow.py) remains unchanged in structure. A new --remove-age1 CLI flag is forwarded from the dispatcher to the year-specific script, where a shared hook in cli_utils.py reads the flag and sets REMOVE_AGE1 at runtime. This means the same hake_<year>.py file can produce a full all-ages run or an age-2+ run from the same CLI contract, controlled entirely by whether --remove-age1 is passed.

../_images/age1_exclusion_workflow_steps.svg

Source files and usage#

Three cooperating source layers implement the age-1 exclusion contract. The dispatcher layer in year_specific_workflow.py parses --year, --verbose, --compare, and the new --remove-age1 flag, then routes execution to the selected year script using runpy.run_path. The shared-hook layer in cli_utils.py defines get_verbose(), get_compare(), and the new get_remove_age1(), which year scripts consume to set VERBOSE, COMPARE, and REMOVE_AGE1 at runtime without touching their source code. The year-specific script layer contains all year-specific file paths and parameters and includes the three conditional age-1 blocks that reference REMOVE_AGE1 and AGE1_DOMINATED_HAULS.

../_images/age1_exclusion_orchestration_flow.svg

Dispatcher CLI arguments#

The dispatcher interface adds one new flag to the existing set:

--year (required): Selects which year script to execute (for example, hake_2019.py or hake_2023.py).
--verbose (optional flag): Enables stage-level logging output in the selected year script.
--compare (optional flag): Enables the EchoPro-vs-Echopop comparison pathway.
--remove-age1 (optional flag): New. Signals the year script to activate the age-1 exclusion branch, which removes haul-level and signal-level age-1 contributions and produces age-2+ estimates.

Because the dispatcher uses parse_known_args(), the --remove-age1 flag is forwarded to the year script without being consumed or dropped. The year script then reads the flag through the get_remove_age1() hook and uses it to set REMOVE_AGE1 before the processing logic begins. An empty AGE1_DOMINATED_HAULS = [] still controls the optional haul filter independently; users who have identified hauls dominated by age-1 fish can populate this list regardless of whether --remove-age1 is set.

Dispatcher CLI design#

The updated dispatcher adds --remove-age1 alongside the existing flags and forwards it to the year script through the same sys.argv forwarding pattern.

The only change from the original dispatcher is three lines: the add_argument declaration for --remove-age1 and the conditional sys.argv.append block that forwards it when the flag is present. All other routing and execution logic is unchanged.

Shared hooks and workflow integration#

Year-specific scripts use shared utility hooks from cli_utils.py to read CLI state at runtime. Adding get_remove_age1() follows the same pattern as get_verbose() and get_compare(), keeping hook semantics consistent across all flags.

This new hook is incorporated into each year-specific script by replacing the hard-coded REMOVE_AGE1 = True or REMOVE_AGE1 = False assignment in the parameter block with a try/except pattern that mirrors how VERBOSE and COMPARE are already handled:

The try/except fallback allows the same year script to run cleanly from the CLI (where --remove-age1 may or may not be present), from a direct Python invocation, and from an interactive REPL session, while still defaulting to a predictable behavior when the flag is absent.

Decision points in the script#

With REMOVE_AGE1 resolved from the CLI hook, the age-1 logic appears in four specific places in the workflow. Every decision point — including the haul filter — must be gated on REMOVE_AGE1 so that an all-ages run (no --remove-age1) leaves the data completely untouched even if AGE1_DOMINATED_HAULS contains values. The snippets below show the correct conditional form for each placement.

CLI usage examples by platform#

In each platform variant below, --remove-age1 is added after the year target. All other flags behave identically to the year-specific workflow contract. Omitting --remove-age1 produces a full all-ages run from the same year script.

Windows CMD

REM Age-2+ only run
python year_specific_workflow.py --year hake_2019 --remove-age1

REM Age-2+ only with verbose logging
python year_specific_workflow.py --year hake_2019 --verbose --remove-age1

REM Age-2+ only with verbose logging and EchoPro comparison
python year_specific_workflow.py --year hake_2019 --verbose --compare --remove-age1

REM Standard all-ages run (no flag needed)
python year_specific_workflow.py --year hake_2019 --verbose

Windows PowerShell

# Age-2+ only run
python year_specific_workflow.py --year hake_2019 --remove-age1

# Age-2+ only with verbose logging
python year_specific_workflow.py --year hake_2019 --verbose --remove-age1

# Age-2+ only with verbose logging and EchoPro comparison
python year_specific_workflow.py --year hake_2019 --verbose --compare --remove-age1

# Standard all-ages run (no flag needed)
python year_specific_workflow.py --year hake_2019 --verbose

Linux/macOS bash

# Age-2+ only run
python3 year_specific_workflow.py --year hake_2019 --remove-age1

# Age-2+ only with verbose logging
python3 year_specific_workflow.py --year hake_2019 --verbose --remove-age1

# Age-2+ only with verbose logging and EchoPro comparison
python3 year_specific_workflow.py --year hake_2019 --verbose --compare --remove-age1

# Standard all-ages run (no flag needed)
python3 year_specific_workflow.py --year hake_2019 --verbose

Notebook-driven orchestration#

A notebook can serve as a lightweight orchestration surface for launching age-1 exclusion runs when live logs, incremental diagnostics, and reproducible command history are needed in one place. The pattern below adds --remove-age1 to the dispatcher command and streams output in real time so stage progress is visible as it occurs.

Shell escape (!)

The shell escape (!) is the most direct method and is suitable for one-off runs where streaming output is not critical.

# Set year
year = 2019

# Age-2+ only run
!python year_specific_workflow.py --year hake_{year} --verbose --remove-age1

Programmatic dispatch

The subprocess module streams log output while the process runs, provides reliable termination on kernel interrupt, and exposes the exit code so failures are immediately visible.

import subprocess

# Parameterize the execution
year = 2019
cmd = [
    "python", "year_specific_workflow.py",
    "--year", f"hake_{year}",
    "--verbose",
    "--remove-age1",
]

print(f"Starting age-2+ workflow for {year}...\n")

# Launch the process with piped streams
process = subprocess.Popen(
    cmd,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,  # Redirect errors to the standard output stream
    text=True,                 # Decode bytes to strings automatically
    bufsize=1                  # Line-buffered for immediate printing
)

# Iterate over the output stream until the process terminates
try:
    for line in process.stdout:
        print(line, end="")
except KeyboardInterrupt:
    process.kill()
    print("\n[Workflow interrupted by user; process terminated]")

process.wait()
if process.returncode == 0:
    print(f"\nAge-2+ workflow for {year} completed successfully.")
else:
    print(f"\nWorkflow failed with exit code {process.returncode}.")

Batch orchestration across multiple years#

When age-1 exclusion needs to be applied across a series of survey years, wrapper scripts provide a practical control layer that reuses the same dispatcher contract and adds --remove-age1 to every invocation.

Windows .bat

To run: Open Command Prompt, navigate to the folder, and type run_hake_age2plus.bat.

@echo off
setlocal enabledelayedexpansion

set YEARS=2015 2017 2019 2021 2023

for %%Y in (%YEARS%) do (
    echo [INFO] Running age-2+ workflow for hake_%%Y ...
    python year_specific_workflow.py --year hake_%%Y --verbose --remove-age1

    if errorlevel 1 (
        echo ❌ Error encountered for year %%Y
    ) else (
        echo ✅ Completed year %%Y
    )
)

Windows PowerShell (.ps1)

To run: Open PowerShell, navigate to the folder, and type .\run_hake_age2plus.ps1.

$years = @(2015, 2017, 2019, 2021, 2023)
$failedYears = @()

foreach ($year in $years) {
    Write-Host "Running age-2+ workflow for year $year..."
    python year_specific_workflow.py --year "hake_$year" --verbose --remove-age1

    if ($LASTEXITCODE -eq 0) {
        Write-Host "✅ Completed year $year" -ForegroundColor Green
    } else {
        Write-Host "❌ Error encountered for year $year" -ForegroundColor Red
        $failedYears += $year
    }
}

if ($failedYears.Count -gt 0) {
    Write-Host "`nSummary of Failures: $($failedYears -join ', ')"
}

Linux/macOS bash (.sh)

To run: Open Terminal, navigate to the folder, and type ./run_hake_age2plus.sh.

#!/usr/bin/env bash
set -u

years=(2015 2017 2019 2021 2023)
failed_years=()

for year in "${years[@]}"; do
  echo "Running age-2+ workflow for year ${year}..."

  if python3 year_specific_workflow.py --year "hake_${year}" --verbose --remove-age1; then
    echo "✅ Completed year ${year}"
  else
    echo "❌ Error encountered for year ${year}"
    failed_years+=("$year")
  fi
done

if [ ${#failed_years[@]} -gt 0 ]; then
  echo -e "\nSummary of Failures: ${failed_years[*]}"
  exit 1
fi

Practical notes#

Choosing between `REMOVE_AGE1` and `AGE1_DOMINATED_HAULS`#

These two parameters address different levels of the exclusion problem. REMOVE_AGE1 removes the age-1 signal from the population estimates by scaling NASC, abundance, and biomass proportionally, and it operates at the stratum level across all transect intervals. AGE1_DOMINATED_HAULS is a haul-level filter that removes specific trawl hauls from both the NASC and biodata tables before any processing begins. The haul filter is appropriate when certain trawls are so heavily dominated by age-1 fish that their presence would distort the proportion tables even after signal removal. The two mechanisms are independent and can be used together, in isolation, or not at all.

Implementing age-1 exclusion in a new workflow#

To add age-1 exclusion to a new year script, add get_remove_age1() to cli_utils.py, add --remove-age1 to the dispatcher, and then add four blocks to the year script: a try/except block in the parameter section to set REMOVE_AGE1 from the CLI hook; the haul-removal guard after ingestion; the age-1 proportion extraction after binning and count tables; and the remove_group_from_estimates call after inversion along with the reallocate_excluded_estimates call after kriging.

Reproducibility note

Always document the AGE1_DOMINATED_HAULS list and whether --remove-age1 was passed alongside your run outputs. Because REMOVE_AGE1 defaults to False in interactive mode, an undocumented run may produce all-ages results even when age-1 exclusion was intended.