Build workflow CLI hooks from simple to advanced

This page details the progression from basic command-line inputs to advanced, modular workflow control. This starts with the smallest useful CLI hook and then expands to configuration files and modular stage routing.

What is CLI and why is it useful

A Command Line Interface (CLI) is a text-based bridge between a user and a script. It allows users to pass runtime arguments (instructions provided at the exact moment a script is executed) to modify software behavior without changing the source code. In Python workflows, this is typically implemented using the argparse library, which handles the passing of terminal strings into Python-accessible variables.

But what is a CLI hook?

A hook is a defined entry point that connects terminal commands to internal code logic. A hook serves as a “handle” on the outside of the software. When a user activates that handle by entering a specific flag, such as --run_algorithm_A, the software catches that signal and executes the corresponding logic branch. Without these hooks, a script acts as a “black box” that performs the same fixed actions every time it runs.

CLI hooks facilitate automation and reproducibility by allowing users to run repeatable commands through shell scripts or batch files. This approach eliminates the need for manual script editing, reduces human error, and creates a clear record of the specific parameters used for every execution run.

Simple hooks

The most basic pattern is a boolean flag. This is a simple “on/off” switch that controls a single, specific behavior within the workflow. It is an ideal starting point for adding user-driven control to an existing process. So drop_specimen_only_hauls is called using conditional logic based on the user’s terminal input.

If there is a biodata ingestion stage in a workflow script called workflow_runner.py, then adding --drop_specimen_hauls can define whether or not to use drop_specimen_only_hauls.

Don’t use drop_specimen_only_haul

Windows

python workflow_runner.py

Linux/macOS

python3 workflow_runner.py

Use drop_specimen_only_hauls

Windows

python workflow_runner.py --drop_specimen_hauls

Linux/macOS

python3 workflow_runner.py --drop_specimen_hauls

In other words, adding --drop_specimen_hauls is effectively equivalent to:

if drop_specimen_hauls is True:
    biology.drop_specimen_only_hauls(dict_df_bio)

This requires inserting the hook into workflow_runner.py.

worfklow_runner.py

import argparse

# Assume that dict_df_bio has already been fully loaded ...
# Create the ArgumentParser object to hold the definitions of expected arguments
parser = argparse.ArgumentParser(description="Run biodata step")

# Define a hook. The 'action="store_true"' parameter ensures that the variable becomes 'True' if 
# the flag is present and remains 'False' if it is omitted
parser.add_argument(
    "--drop_specimen_hauls",
    action="store_true",
    help="Drop specimen-only hauls after biodata ingestion"
)

# Convert the terminal strings into an 'args' object with properties corresponding to the defined 
# hooks
args = parser.parse_args()

# Execute conditional logic based on the user's terminal input.
if args.drop_specimen_hauls:
    biology.drop_specimen_only_hauls(dict_df_bio)

The implementation of a CLI hook relies on three primary phases: initialization, definition, and parsing.

1. Initialization
   The argparse.ArgumentParser object serves as the container for the entire command-line interface. By setting a description, the script provides context that appears when a user views the help documentation. This object is responsible for managing the rules and expectations for any data passed from the terminal.

2. Defining the hook
   The add_argument method is where the specific “hook” is established. By defining --drop_specimen_hauls, the script creates a recognized handle for the user.

   • The flag: The name starting with double dashes (--) indicates an optional parameter.

   • The action: Using action="store_true" tells Python that this is a binary switch. If the user includes the flag, the internal variable is set to True; if they omit it, the variable defaults to False.

   • The help: The help string is stored to provide automatic documentation for the user.

3. Parsing terminal input
   The parse_args() method is the execution step that reads the strings provided in the terminal. It compares those strings against the defined hooks and packages the results into a simple object (typically named args). This allows the rest of the script to access the user’s choices using standard Python “dot notation,” such as args.drop_specimen_hauls.

Hooks as a repeated utility

Once a basic boolean hook is functional, it can be further modified to improve reusability.

If multiple workflow scripts require the same behavior (e.g., --drop_specimen_hauls toggle), copying and pasting argparse code into every file increases technical debt and leads to “logic drift.” A more robust pattern involves moving repeated parser logic into a shared utility module. This establishes a single source where any change to a flag name or documentation automatically propagates across all workflows. This centralized approach provides several key advantages:

• One definition: the flag name, action, and help text are defined once.

• Consistent behavior: every script interprets the hook the same way.

• Easier maintenance: updates happen in one place rather than many scripts.

The schematic below illustrates the relationship between the shared utility and the individual workflow scripts.

For instance, there may be two workflows (workflow_alpha.py and workflow_beta.py) that both use --drop_specimen_hauls in the same way. Their respective directory can then be updated by adding the shared hook utility file cli_hook_utils.py:

shared_workflows/
  workflow_alpha.py
  workflow_beta.py
  cli_hook_utils.py

So rather than defining a separate argparse.ArgumentParser instance for each workflow file, cli_hook_utils.py can instead inject the hooks into any target file.

cli_hook_utils.py

# CLI hook utilities file [cli_hook_utils.py]
import argparse

def add_drop_specimen_hauls_arg(parser: argparse.ArgumentParser) -> argparse.ArgumentParser:
    parser.add_argument(
        "--drop_specimen_hauls",
        action="store_true",
        help="Drop specimen-only hauls after biodata ingestion",
    )
    return parser

def should_drop_specimen_hauls(args: argparse.Namespace) -> bool:
    return args.drop_specimen_hauls

These utilities are then used in each of the example workflow files:

workflow_alpha.py

import argparse
from cli_hook_utils import add_drop_specimen_hauls_arg, should_drop_specimen_hauls

parser = argparse.ArgumentParser(description="Run workflow alpha")
parser = add_drop_specimen_hauls_arg(parser)
args = parser.parse_args()

if should_drop_specimen_hauls(args):
    biology.drop_specimen_only_hauls(dict_df_bio)

workflow_beta.py

import argparse
from cli_hook_utils import add_drop_specimen_hauls_arg, should_drop_specimen_hauls

parser = argparse.ArgumentParser(description="Run workflow beta")
parser = add_drop_specimen_hauls_arg(parser)
args = parser.parse_args()

if should_drop_specimen_hauls(args):
    biology.drop_specimen_only_hauls(dict_df_bio)

The run behavior also remains consistent across the individual workflow files:

Windows Command Prompt

python workflow_alpha.py --drop_specimen_hauls
python workflow_beta.py --drop_specimen_hauls

Windows PowerShell

python workflow_alpha.py --drop_specimen_hauls
python workflow_beta.py --drop_specimen_hauls

Linux or macOS bash

python3 workflow_alpha.py --drop_specimen_hauls
python3 workflow_beta.py --drop_specimen_hauls

Parameterized hooks and configuration routing

While a boolean flag like --drop_specimen_hauls is a binary “switch,” a parameterized hook acts as a container or a pointer. It tells the script not just what to do, but where to look or what specific value to use.

Species filter

Incorporate a species-specific filter that includes only the provided identifier.

python workflow_runner.py --species_id 22500

TS-length regression

Provide TS-length regression coefficients.

python workflow_runner.py --ts_length_params slope 20 intercept -68.0

Jolly and Hampton (1990) algorithm

Provide multiple parameter values required for the Jolly and Hampton (1990) stratified random sampling algorithm.

python workflow_runner.py ^
    --jollyhampton transects_per_latitude 5 ^
                    transect_proportion 0.75 ^
                    num_replicates 1000

Note: The ^ is a line continuation character for a Windows Batch (.bat or cmd.exe) script.

NASC file

Specify the filepath and sheet name of an Excel file containing integrated acoustic backscatter (i.e., \(S_\text{A}\)).

python workflow_runner.py ^
    --data_root C:/Data/EchopopData/ ^
    --nasc_file nasc_data_2030.xlsx ^
    --nasc_sheet Sheet1

Note: The ^ is a line continuation character for a Windows Batch (.bat or cmd.exe) script.

As workflows scale, entering dozens of individual parameters into a terminal becomes cumbersome and error-prone. User-generated configuration files resolve this by centralizing all paths and settings into a single document, or a set of modular files. This shifts the CLI’s role from a list of granular settings to a router. A single parameterized hook called --config points the script toward a configuration file where logic, such as the --drop_specimen_hauls toggle, is embedded directly. In the case of biodata processing, this ensures that the conditional logic for data ingestion remains bundled with the data it describes. This means cli_hook_utils.py can be further generalized to accommodate workflow-level configurations.

cli_utils.py

# CLI hook utilities file [cli_hook_utils.py]
import argparse
from pathlib import Path
import json
import yaml

def add_config_arg(parser: argparse.ArgumentParser) -> argparse.ArgumentParser:
    """Adds a mandatory --config flag for passing external settings files."""
    parser.add_argument(
        "--config",
        required=True,
        help="Path to configuration file (.yaml, .yml, or .json)",
    )
    return parser

def load_config_params(config_path: str | Path) -> dict:
    """Parses YAML or JSON files into a Python dictionary."""
    path = Path(config_path)
    suffix = path.suffix.lower()

    if suffix in {".yaml", ".yml"}:
        return yaml.safe_load(path.read_text(encoding="utf-8"))
    if suffix == ".json":
        return json.loads(path.read_text(encoding="utf-8"))

    raise ValueError(f"Unsupported extension '{suffix}'. Use .yaml or .json.")

This updates the CLI command to:

Windows Command Prompt

python workflow_runner.py --config configs/biodata_ingestion.yaml

Windows PowerShell

python workflow_runner.py --config configs/biodata_ingestion.yaml

Linux or macOS bash

python3 workflow_runner.py --config configs/biodata_ingestion.yaml

---

Configuration-driven execution

In this advanced pattern, the workflow script delegates parameter handling entirely to the configuration file. This means that the CLI no longer needs a --drop_specimen_hauls flag; the preference is read directly from the loaded dictionary. Using the configured biodata ingestion workflow example, config_params is instead defined by the argparse.ArgumentParser:

JSON configuration

# Import packages
from pathlib import Path
from echopop import ingest
import json
from echopop.survey import biology

# Read in configuration
config_path = Path("configs/biodata_setup.json")
config_params = json.loads(config_path.read_text(encoding="utf-8"))

YAML configuration

# Import packages
from pathlib import Path
from echopop import ingest
import yaml
from echopop.survey import biology

# Read in configuration
config_path = Path("configs/biodata_setup.yaml")
config_params = yaml.safe_load(config_path.read_text(encoding="utf-8"))

New CLI approach

# Import packages
from pathlib import Path
from echopop import ingest
from cli_hook_utils import add_config_arg, load_config_params # <---- Note new module import
from echopop.survey import biology

# Initialize parser with shared configuration hook
parser = argparse.ArgumentParser(description="Run biodata ingestion")
parser = add_config_arg(parser)
args = parser.parse_args()

# Load the "contract" from the file pointer
config_params = load_config_params(args.config)

This approach is useful because a single CLI entrypoint (--config) manages complex runs. It further provides reproducibility for parameters like drop_specimen_hauls that are stored in the configuration file, creating a (semi-)permanent record of how the data were processed.