Skip to content

Latest commit

 

History

History

spec

README.md

Machine-readable specification of the phyphox file format

This directory is a probe, not a commitment. It contains one block of the format — input — modeled completely, to find out what phase 3 actually costs before ~540 attributes are written in the same shape. FINDINGS.md has the result.

Nothing consumes these files yet. They are not wired into the docs build.

Why a schema rather than RELAX NG

A grammar can say that rate is a float. It cannot say that rateStrategy defaults to auto from file format 1.14 and to limit below it, which is real behavior in both parsers (InputElementHandler.swift, SensorInputDescriptor.defaults(forVersion:); PhyphoxFile.java, inputBlockParser). Version-dependent defaults are the reason this is structured YAML that generates a validator, rather than a validator written by hand.

The RELAX NG grammar exists anyway — as a product: tools/generate_validators.py derives docs/assets/validators/{phyphox.rng,phyphox.rnc,phyphox.sch} from these files on every build, and tools/hooks.py (_check_validators) proves the result against the corpus and the shipped collection before the site ships it. The Schematron carries what the grammar cannot express — buffer references, container-name uniqueness, the graph dataset pairing, the mapColor[N] name shape, and the version gates as warnings. Two traps met while writing the generator, so nobody re-finds them the hard way: libxml2 validates a plain run of optional attributes by backtracking and blows up exponentially once ~16 are present on one element (wrap each element's attributes in an interleave, which takes its linear attribute-group path), and it hangs compiling wide element interleaves and repeated inline <data> facets (merge all-optional children into one repeated choice; share value types as named defines). The grammar is deliberately no stricter than the parsers: stray text and foreign-namespace attributes are admitted everywhere, because the apps skip both.

Shape

One file per block. Each element:

elements:
  - name: sensor
    since: "1.0"
    parent: input
    summary: One-line description.
    attributes:
      - name: stride
        type: integer
        default: 1
        summary: ...
        agreement: agreed          # see below
    children: [output]
    outputs:                       # components a sensor may map to a buffer
      - {name: x, summary: ...}

parent: may be a list when the same element is legal under several parents with the same meaning - the view elements sit under view and under every view group (vertical, horizontal, grid, stack, transform). Every loader registers the entry once per parent, so (block, parent, name) stays the key and {{spec:views/view/graph}} keeps resolving; the RELAX NG gets one define per pair. Recursion (a group under a group) is just another entry in the list.

A container element declares the attributes all its children accept with child_attributes: [group, ...], naming groups under the block's common:. view and the view groups declare view_element_attributes (label, visibility); horizontal additionally declares horizontal_child_attributes (weight). The build fails if a named group does not exist. This replaces the former hard-coded "every child of view accepts label and visibility"; the analysis block's module_attributes still use the older hard-coded path.

An attribute whose name is numbered without bound - the map graph's mapColor1, mapColor2, ... - is modelled once, with name_pattern: holding a regular expression; the validators accept every attribute matching it and the name: (e.g. mapColor[N]) is only what the rendered documentation displays.

Defaults that depend on the format version are a list, most recent first; the first entry whose since the document satisfies wins:

    default:
      - since: "1.14"
        value: auto
      - value: limit

units.yml is a table, not a block

spec/units.yml holds no elements. It is the table of known units behind the unit attribute type (value/@unit, edit/@unit, the graph's unitX/Y/Z/YperX): id, symbol, quantity, conversion to the quantity's base unit, system and the declared counterpart the app's unit-system setting switches to. docs/file-format/units.md renders it through the {{units}} marker, generate_validators.py derives the unit-reference Schematron rules from it, validate_experiments.py checks @id values against it, and the apps and the remote interface copy the numbers. tools/hooks.py (_check_units) checks the table's shape and, with the app checkouts beside this repository, that the string tables carry common_unit_short_<id> for every unit that has a placeholder form (all of them once the table's agreement is flipped to agreed). The loaders of the other tools read a spec file for elements: and common: only, so a file with neither is inert to them.

agreement is mandatory on every attribute

This is the field that keeps the spec honest. It records what was found when the two parsers were compared, and it is what makes the spec reviewable rather than merely plausible:

value meaning
agreed Android and iOS behave the same; the spec states that behavior.
divergent They differ. Requires inconsistency: naming an entry in ../inconsistencies.yml. The spec states the behavior only if that entry has been decided.
undecided They differ and nobody has chosen. Also requires inconsistency:.
platform Deliberately one-sided, because the feature only exists on one platform. Requires platforms:.
planned Specified ahead of the implementations: the spec is the design, and no app reads it yet. Rendered with a "planned, not yet implemented" note. Flip to agreed once both apps ship it; a divergence found on the way is recorded like any other.

Where the entry has been decided, the attribute also carries rules: naming the rule in rules.yml that settles it. rules.yml holds the questions that turned out not to belong to any one block — how to treat an invalid enumerated value, whether enum matching folds case, whether output components are validated. Answering those once is what keeps the decision count from growing with the attribute count.

The documentation is a third opinion, and it is checked

tools/spec_vs_docs.py diffs the spec against the documentation pages and runs as part of mkdocs build --strict. Anything the docs describe that the spec does not model fails the build. The other direction is reported but tolerated — the docs are often behind, and being behind is not a modeling error.

The prose is not a source: it has been caught three times naming an attribute no implementation accepts (autoExposure, showControls, logy). Its value is as an independent list of what exists.

Child elements are checked, not trusted

tools/hooks.py fails the build if an element declares a child in children: that is not modeled, or models an element whose parent does not list it. Both directions matter: the graph data picker and the button's trigger tag were both lost by writing a name into children: and stopping there.

That check cannot tell you a child exists in the first place — only the parsers can. See below.

Enumerate child elements from both parsers, not one

The first pass at views.yml gave graph the single child input, missing output entirely — and with it the whole data-picker feature. iOS registers both (childHandlers = ["input": ..., "output": ...]) and Android parses both; the omission was simply not looking. Children deserve the same treatment as attributes: list them from each parser and compare, rather than assuming the obvious ones are all there are.

The same pass filed calibrationParameter under input when it belongs to output. When a handler declares several private Attribute enums, check which one each belongs to.

A reliable way to enumerate them on iOS is the childHandlers dictionary each handler builds; on Android it is the AdditionalTag names each view element accepts from ioBlockParser, which are checked with an explicit at.name.equals(...) chain ending in "Unknown tag". Reading those two lists side by side gives the child set directly.

Extracting a list of cases from a parser

Two independent traps, both hit while probing the analysis modules:

  • Bound the class by brace depth, not by a byte window. analysisBlockParser ends at line 3472; a fixed-size window ran past it and picked up audio, bluetooth, data, flashlight and set from the parsers that follow.
  • A case "..." is not necessarily an element or module name. Inside the same class, nested switches match attribute values: sum is a value of the map module's zMode, and linear, nearest, next, previous are values of interpolate's method. Indentation separates the two levels reliably here; context (= new, Mode.) confirms it.

Of twelve names that first looked like Android-only analysis modules, exactly two were — butterworth and imagedecode, both file format 1.20.

Text content is part of the format

Many child elements carry their meaning in their text rather than in an attribute: an <output> names a buffer, a <trigger> names an id, a <map> carries the label to display. Record that with text:. A model that only lists attributes silently loses it — the dropdown's option labels were briefly modeled as a replacement attribute, which iOS declares but never reads and which appears nowhere in the documentation.

Read the prose under a definition-list term, not just the term

Twice a platform limitation was already documented and still recorded as an unexplained divergence, because the note lives in the paragraph below the term rather than in the : lines beside it — depth/smooth ("Only applies to LiDAR on iOS devices") and info/wifiSignalStrength ("Only available on Android."). tools/spec_vs_docs.py compares names only and cannot catch this; the description has to be read.

Android has two parsers; iOS has one

PhyphoxFile.java is Android's full parser, used when an experiment is opened. A separate, minimal parser under ExperimentList/datasource/ builds the collection list. iOS uses one parser for both. So an attribute that only matters before an experiment is opened may be absent from PhyphoxFile.java by design — isLink, which marks a collection entry that redirects to a web page rather than running, is implemented only in the list parser, because the full parser never sees such a file.

That is a third reason an attribute can be missing from the parser you are reading, after dynamic names and attributes read by the class that consumes them.

Verifying a one-sided attribute

Before recording an attribute as read by only one app, check how the other parser names attributes. Android builds the map-graph color stops as "mapColor" + index in a loop, so mapColor1 appears nowhere in its source and a grep for it wrongly suggested the attribute was iOS-only. That is currently the only dynamically constructed attribute name in either codebase, but the failure is silent, so a negative grep is not on its own evidence of a divergence.

platform must not become a dumping ground for divergences nobody wants to think about. An attribute one parser silently ignores is divergent, not platform, unless there is a reason it cannot exist on the other side.

What the generated validators do not catch, and why

The iOS test suite used to carry 34 invalid fixtures that only it ran. They are in the shared corpus now (2026-08-29), so both apps assert every rule they pin; what follows is the account of what the published RELAX NG and Schematron could and could not be taught along the way.

They cover 56 of the 59 files in corpus/invalid. Eight rules were modelled during the move, each read off the spec so the generator follows automatically:

field on catches
required: base-locale title, category an experiment with no name for its own language
content_required: true root link a link that points nowhere
unique_among_siblings: true link/@label, root and translation two links sharing the key they are matched on
exclusive_minimum: 0 gausssmooth/@sigma, both audio/@rates a zero-width gaussian, a negative sample rate

plus one relationship rule written out rather than derived: a translated <link> with no URL must carry the label of a root link, whose URL it inherits.

required: base-locale is not required: true, and the build is what said so. The strict version failed against light.phyphox in the shipped collection, which has no root <title> at all and takes it from <translation locale="en"> with no root locale set.

The three the grammar cannot reach

missing-required-slot, subtract-one-input-of-two and value-not-allowed-for-slot are in VALIDATOR_BLIND - the first entries it has ever had, and exactly the case it was written for. An <input> without an as attribute belongs to whichever slot the apps' positional matching gives it, and no grammar can redo that. validate_experiments.py catches them by asking the two questions that do not need to know which slot: if no slot of the module takes a literal then no input may carry one, and slots demanding three tags between them cannot be satisfied by two.

Where the checks live

Three callers run the spec over a file - the CLI, the docs build, and the BLE lab's capture check - and each once carried its own copy of the sequence. A check added to one silently did not exist in the others, twice in one afternoon. They all call validate_experiments.check_file() now. If a fourth appears, it calls that too.