Workflow Automation Documentation

Overview

What Workflow Automation provides

Workflow Automation combines dependency-aware experiment scheduling with scientific data management, data processing, visualization, optimization and quality control.

Before you begin

Install and activate SIGCLEAR, then run sglicense status to verify your license.

Quick start

Start a simple experiment in SIGCLEAR

Create an empty project directory and add a file named SConstruct:

from experiment import *

Process('raw', None,
    '''
    sgcreate nx=10 data.ns=500
    | sgfieldmath offset:f="50*ix"
      data="sinc(30*(data.time-(600^2+offset^2)^0.5/1500))"
    ''')

Process('processed', 'raw',
    'sgfieldmath data="2*data"')

Figure('./processed.png', 'processed',
    'sgplotps')

Run the experiment from that directory:
```
scons
```

The workflow creates the dataset data/raw.sg, transforms it into data/processed.sg, and generates processed.png as a quality-control figure.

Core concepts

Process, Program and Package

As shown in the Quick Start, a SIGCLEAR experimental workflow is defined in a Python-based SConstruct file using the SIGCLEAR Experiment API. Three concepts organize the workflow: Process, Program and Package.

Process

A workflow is composed of Processes. Each Process defines one or more output datasets, its input datasets, and the processing flow that transforms the inputs into outputs.

A complex experiment can contain many Processes connected in a dependency graph. When an input, parameter or Program changes, SIGCLEAR rebuilds the affected datasets and regenerates their dependent figures. Unaffected Processes and figures are not rerun.

The Quick Start contains three Processes. The first creates raw; the second reads raw, doubles its data vector, and writes processed; the third generates processed.png from processed:

raw → processed → processed.png

Figure is a specialized Process

Within the workflow model, Figure is a specialized Process for visual quality-control output. It participates in the same dependency graph as other Processes.

A workflow imports the SIGCLEAR Experiment API with from experiment import * before defining Processes and Figures.

Program

A Program is an executable that performs a specific operation within a Process. A processing flow can run one Program or connect several Programs through a pipeline to transform input datasets into output datasets.

To create raw, the first Process runs sgcreate to create a dataset containing a 500-sample data vector across 10 columns. It then runs sgfieldmath to add the offset field and calculate the sinc signals stored in data.

The second Process runs sgfieldmath to double the data vector. The Figure runs sgplotps to generate the plot.

SIGCLEAR Programs can also run directly from the Linux command line. For example, this command doubles the data vector in the raw dataset:

sgfieldmath data="data*2" <data/raw.sg >data/processed-cli.sg

Program manuals

Use man with a SIGCLEAR Program name to open its command reference:

man sgcreate

Package

A Package is an installable collection of related Programs, libraries, modules and supporting resources. Each Package provides focused capabilities for experimental workflows.

sigclear-experimentExperiment API and process scheduling sigclear-soigLarge-scale multidimensional dataset management sigclear-ioConversion between SIGCLEAR datasets and other data formats sigclear-dspSignal synthesis, filtering and analysis sigclear-plotVisualization and experiment QC sigclear-optimOptimization and parameter inversion

In the Quick Start, sigclear-experiment provides the Experiment API and execution scheduling. The Program sgcreate is provided by sigclear-soig; sgfieldmath and sgplotps are supplied by sigclear-dsp and sigclear-plot, respectively.

Users can install additional Packages or develop custom Packages to add specialized functions to their experimental workflows.

Common Program options

Control shared SIGCLEAR Program behavior

Options beginning with two hyphens control behavior shared by SIGCLEAR Programs. Program-specific options remain documented in each Program manual.

Option	Purpose
`--help`	Open the Program's installed manual and exit before processing input.
`--default=VECTOR,...`	Select the default vectors processed by Programs that operate on dataset vectors.
`--nthread=N`	Set the OpenMP thread count for Programs that support parallel processing.
`--verb=y\|n`	Enable or suppress processing and data-I/O progress messages.
`--hist=append\|ignore\|clear`	Append the current command to dataset history, preserve history without appending, or clear history.
`--blksz=N`	Set the input file-buffer size in dataset columns. This advanced option can reduce memory use when reading datasets with large columns or groups.

For example, the following command fits only the pressure and altitude vectors and limits supported parallel work to four threads:

sgfit --default=pressure,altitude --nthread=4 \
    <observations.sg >prediction.sg

Exact option names

The thread option is --nthread, singular. These command-line options are separate from similarly named options accepted by the Python Process and Figure APIs.

Examples

Follow complete research workflows

NASA air-pressure fittingImport atmospheric measurements and compare robust model fits Lunar LPR data analysisImport instrument records, filter radar channels and generate quality control

Next steps

Move from example to your research workflow

Request a trial Explore implementation services Discuss your workflow

Build a reproducible scientific workflow.