sigclear-soig

Overview

Scientific and engineering data management

sigclear-soig provides the dataset model, libsoig APIs and Programs for creating, inspecting and reorganizing scientific data within Workflow Automation.

Dataset structure

One schema, many measurement records

The header stores processing history, dataset-level attributes and a field schema. Every column contains values described by that same schema. Groups are consecutive runs of columns, rather than separate field definitions for each group.

A column is not necessarily one numerical sample or one CSV row. A scalar-only dataset may represent one observation per column; a trace dataset can contain hundreds of samples within each column. Different vector fields may have different lengths, but each field's declared length applies to every column. Text fields have a fixed byte width.

seed.sg
├── Header
│   ├── Attributes: data.ns=100, data.s0=0, data.ds=0.001
│   │               group key → group field; index key → index field
│   └── Field schema: group (integer scalar), index (integer scalar),
│                    data (100 floating-point samples)
└── Columns: one group with group-key value 0
    ├── Column 0: group=0, index=0, data=[0, ..., 0]
    └── Column 1: group=0, index=1, data=[0, ..., 0]

Grouping rules

Choose the processing boundary explicitly

The group attribute identifies the scalar field used as the group key. Values in that field determine membership; the field need not itself be named group. The subgroup and index attributes likewise identify fields used by Programs that need subdivisions or indexing.

Group reads follow adjacent columns, not a global SQL-style group-by:

Column key values:  1, 1, 2, 1
Consecutive groups: [1, 1] [2] [1]  → three groups, not two

Decide which measurements belong together, arrange their order, and inspect the key before group-wise processing such as sgstack. Sorting or concatenating can change group boundaries.

Sampling and units

Keep coordinates consistent with the vector

For a regularly sampled vector named data, these header attributes describe its sampling:

The coordinate at zero-based sample index i is data.s0 + i * data.ds, for i=0,...,data.ns-1. The example spans 0 through 0.099, not 0.100. If the coordinate is time in seconds, the interval is 1 ms.

Use the vector's own name for other sampling attributes, such as pressure.ns, pressure.s0 and pressure.ds. These are common metadata, not extra samples or per-column measurements.

Record physical units, field meanings and missing-value conventions explicitly. Field names do not establish units, and a zero-filled vector is not automatically missing data. Check required types, lengths and spacing in each Program's manual; recheck sampling metadata after windowing, resampling or concatenation.

Quick example

Create two columns containing sampled vectors

sgcreate nx=2 data.ns=100 data.s0=0 data.ds=0.001 </dev/null >seed.sg
sgattribute <seed.sg
sgdump fields=group,index ncol=2 <seed.sg
sgdump ncol=1 <seed.sg

With the current implementation, the creation command produces two columns in one group, each with a zero-filled, 100-sample data vector. Explicit </dev/null selects new-dataset creation rather than transformation of an input dataset.

sgattribute shows history, attributes and field definitions. The first sgdump prints the scalar keys; the second prints vector values from one column. Its itime column is a zero-based sample index, not a physical time coordinate.

Inspect and exchange

Validate the schema before processing

• Use sgattribute to check field names, types, element counts, byte offsets and grouping/sampling attributes.
• Use sgdump to inspect representative values and key sequences. Its diagnostic text is not a replacement binary dataset or a general-purpose interchange export.
• Keep binary outputs redirected to a file or another Program; do not display them directly in the terminal.
• Retain raw inputs, converter parameters, physical units and acceptance checks in the experiment definition.

SOIG uses a native binary representation. Do not assume byte-order or architecture/ABI independence when moving files between systems. For external tools, select the appropriate I/O conversion Program and verify the resulting types, shapes and coordinates on a small sample.

Use sgloadcsv/sgoutcsv for supported table layouts, sgloadmat/sgoutmat for MATLAB exchange, and sgloadbin for explicitly described fixed-size binary records. Choose conversion parameters from the matching manual; conversion does not establish units or correct grouping automatically.

Processing considerations

Structure supports processing; the Program defines behavior

Programs

Choose a Program

Open the installed reference with man PROGRAM, for example man sgcreate, or choose a version and Program from the menu when published. Choose the version matching the installed package; a manual alone does not establish released availability.

Libraries and APIs

Access datasets with libsoig

The Package installs libsoig and its public soig.h interface. Custom Programs use the API to read and write groups, columns, fields, attributes and processing history consistently.

Scope

Numerical research data, not business records

SOIG is designed for numerical arrays and measurements produced by sensors, simulations and scientific algorithms. A relational database remains more suitable for customers, orders and other transactional records.