Grammar Production Coverage¶

Overview¶

Grammars: KerML.xtext, KerMLExpressions.xtext, SysML.xtext from the SysML v2 Pilot Implementation at release 2026-05 — the pin in scripts/pilot-pin.sh, the same one the training corpus and the pilot differential use Provision: ./scripts/download-pilot-grammars.sh (sparse clone at the pin; writes build/pilot-grammars/, not vendored) Run: go run ./cmd/grammar-coverage (writes build/grammar-coverage/) Baseline: grammar-coverage-baseline.json — the counts and the gaps of the last committed run, so a later run can be diffed against it. The full per-production table with citations is not committed: it is build/grammar-coverage/grammar-coverage-tables.md, regenerated by the command above Status: advisory only — nothing gates CI on these numbers, and the tool only reads the grammars and the corpora

The number below is an over-approximation, and it is not a coverage figure. What it measures is input presence: for each production it asks whether some file we already parse contains the terminal literals a path through that production needs. A literal in a file means an input could have driven the production; it does not prove our parser took that path, and it does not prove we handled it correctly. Nothing here says anything about specification compliance — a production with evidence can still be parsed wrongly, and a production without evidence can be perfectly implemented. The honest reading is the negative one: where there is no evidence, no input has ever put us in a position to be right.

We already claim that the whole bundled stdlib, the 100-file training corpus, and the OMG examples and validation corpora parse clean. That is a claim about files. This page is the claim about the grammar: which of its 727 productions those files ever put in front of us.

The denominator¶

Every rule, fragment, enum and terminal declared in the three pinned grammars, extracted by cmd/grammar-coverage (xtext.go parses the Xtext dialect the OMG grammars actually use: @Override, syntactic predicates, actions, cross-references, cardinalities and assignments). Terminals are counted, though literal search can never decide one.

Grammar	Productions	evidence	indistinguishable
`KerML.xtext`	176	117	59
`KerMLExpressions.xtext`	108	39	69
`SysML.xtext`	443	327	116
total	727	483	244

So: 483 of 727 productions have input-presence evidence, 244 are indistinguishable to this method, and 0 have none — against main at 732e8abe654b23475595c41dcdd16efae204ce29, the branch point of the change that added this page. The corpora searched, and what each contributed, are in the baseline and in the generated report.

The buckets¶

evidence — one corpus file contains all the literals of some path through the production. The report cites that file and the line each literal was first seen on.
no-evidence — no single file does. The report says which of the required literals occur nowhere at all, versus occur only in different files.
indistinguishable — literal search cannot decide it, and it is reported as its own bucket rather than folded into either of the others. Three kinds land here: lexer terminals (NAME, STRING_VALUE, matched over characters); pure delegation rules, where every path can be taken without spelling a literal (ConditionalExpression is just NullCoalescingExpression unless you write if … ? … else …); and Identification-style fragments whose content is a name.

Why `no-evidence` is empty, and what to read instead¶

An empty no-evidence bucket is the real result, not an error: of the 217 distinct literals the three grammars use, all but two (% and disjoining) occur somewhere in the corpora, and a production is credited as soon as its cheapest path is present. Since optional groups are, by definition, skippable, the cheapest path of most productions is the one that omits them — which makes the production-level bucket a weak instrument.

So the measurement is refined one level down, to forms: each literal-bearing alternative and optional group inside a production, taken together with the literals every path through that production needs anyway. Disjoining is evidence because disjoint A from B; occurs in the stdlib, but its named form (disjoining D disjoint A from B;) is a form nobody has fed us.

Grammar	Forms	unseen
`KerML.xtext`	273	4
`KerMLExpressions.xtext`	71	1
`SysML.xtext`	463	0
total	807	5

802 of 807 forms have evidence; 5 do not. Those five are the answer to "where are we untested?", and they are adjudicated below.

Adjudicated: the notation nobody has fed us¶

Each entry was checked by hand against the grammar source and against our parser. The verdicts are observations; nothing here was fixed in the change that added this page.

1. `%`, the remainder operator — implemented, never exercised¶

KerMLExpressions.xtext:267 MultiplicativeOperator : '*' | '/' | '%'. % appears in no corpus file and in no fixture; * and / are cited in the stdlib. We do implement it: lexer.Percent (internal/core/lexer/lexer.go) → binaryOpForToken (internal/core/parser/expr.go) → ast.OpMod, and sysml -e '7 % 3' answers 1.

Verdict: real notation, want a fixture. The cheapest gap on this list: an arithmetic operator with no test input anywhere.

2. Prefix metadata on a namespace — implemented, never exercised¶

KerML.xtext:119 Namespace : ( ownedRelationship += PrefixMetadataMember )* NamespaceDeclaration NamespaceBody. Both # and namespace occur in the corpora, never in one file. We implement the prefix (parseMember / the prefix-metadata loop in internal/core/parser/namespace.go), and metadata def Meta; #Meta namespace N { part def A; } analyses clean.

Verdict: real notation, want a fixture.

3–5. KerML's named relationship declarations — not implemented¶

Three forms, all the same shape: a KerML relationship written as a top-level element with its own name, rather than as a clause of a declaration.

Form	Grammar	Status here
`disjoining D disjoint A from B;`	`KerML.xtext:426` `Disjoining`, optional `'disjoining' Identification?`	`disjoining` occurs in no corpus file, and `internal/core/lexer/keywords.go` is the only place it appears in our code
`conjugation Cj conjugate C ~ A.b;`	`KerML.xtext:408` `Conjugation`, the `OwnedFeatureChain` alternative	not parsed
`redefinition R redefinition g :>> f;`	`KerML.xtext:712` `Redefinition`, the `':>>'` alternative	not parsed

All three are rejected by parseMember (internal/core/parser/namespace.go) with expected a namespace member. What we do implement is the clause spelling of the same semantics — the anonymous disjoint X from Y; statement (parseBodyMember, internal/core/parser/defusage.go), ~ conjugation in a typing, and :>> in a usage declaration — so the gap is the standalone, nameable relationship element, which is KerML notation with no SysML v2 surface. specialization S subtype A specializes B; is rejected the same way, which literal search did not flag: specialization occurs in the KerML corpus, so the form counts as seen even though we reject it — a good illustration of what this metric is worth.

Verdict: real notation, not implemented, no fixture. This belongs with the KerML gap analysis, not with a parser change made during a measurement.

What the numbers do not mean¶

Beyond "presence is not execution", the specific ways this over-approximates:

Position is ignored. A literal counts wherever it occurs in the file. then in one transition is evidence for every form that needs then.
Punctuation is a substring match. < is credited by a file containing only <=, and . by any qualified name. Word-shaped literals are matched as whole identifiers, so to is not credited by total.
Comments, doc bodies, string values and quoted names are stripped before the search (stripModelSource), so prose that mentions a keyword is not evidence. This is why hastype is cited from a KerML example rather than from BaseFunctions.kerml, where it is a quoted name.
A path is a lower bound. Optional groups and * cardinalities contribute nothing to a path's requirement, and alternatives contribute their cheapest branch.
Recursion is cut. A path that re-enters the production it came from is not a path; the base case's literals are what get required.
The literal budget is bounded. Path enumeration keeps the 24 cheapest literal sets per production, which can only make a requirement cheaper, never stricter.

Recommended follow-up: instrument the parser¶

A true figure — which productions our parser actually entered, on which inputs — is not reachable by any amount of literal searching. It needs counters in the parse functions and a mapping from our hand-written recursive-descent functions to the OMG productions they implement, then a run over the same corpora. That is a change to every parse function, which is why it was deliberately left out of this measurement, and it is the only honest way to replace the number above with a coverage number.

Until then, the two useful things this harness gives are the adjudicated list above and the committed baseline: re-running it after a parser or corpus change shows which forms moved.

Reproducing¶

./scripts/download-pilot-grammars.sh   # writes build/pilot-grammars/ at the pin
go run ./cmd/grammar-coverage          # writes build/grammar-coverage/: full JSON, tables, text summary
go test ./cmd/grammar-coverage/...     # extractor and classifier unit tests

# refresh the committed baseline
go run ./cmd/grammar-coverage -baseline docs/project/grammar-coverage-baseline.json

The output is deterministic: sorted literals, grammars in file-name order, productions in declaration order, and citations from the first file in a fixed root order. Two runs are byte-identical, which is what makes the committed baseline diffable.

No CI gate. Like the pilot differential, this is advisory: a coverage number that failed a build would be re-baselined into meaninglessness. The figures are also a snapshot of one branch point — the in-flight work on bare import warnings, over-acceptance, reserved words, the KerML comparison and the missing productions will all move them.