Infrastructure · Verification
MetamorphicTesting
Test a stated relation among the outputs of several runs, which needs no exact expected answer for any single run
Definition
Mechanism
Instead of checking each run against a known answer, related runs are compared: a relation states how the outputs must correspond when the inputs are changed in a stated way, as permuting the rows fed to an order-independent aggregation must leave its result unchanged. A relation may involve more than two runs. Each relation is derived, before any run, from a property the correct output must have, stated in the Spec or known of the domain, never from the implementation's behaviour, and it states how a violation is decided: exactly, within a tolerance, or by a statistical test where outputs are random. Follow-up inputs are generated from source inputs, and a violation is a failure on that set of inputs. A satisfied relation shows the outputs are consistent under that relation, not that any single output is correct. It builds tests for EmpiricalTest that need no exact expected answer, including where the correct output cannot be computed independently.
Invariants
- Each relation is derived from a stated property of the correct output, from the specification or the domain, and recorded before execution together with how a violation is decided.
- A violation is reported with the source inputs, the transformations and every output the relation involves.
- A pass is reported as consistency under the named relations, never as correctness.
Risks
Failure modes
Weak relation
the relation also holds for wrong implementations, for example one any constant output satisfies, so passing proves little.
Relation read off the code
the relation is derived from the implementation's behaviour, so the test cannot disagree with it.
Shared error
the defect changes every related output in the same way, so the relation holds.
Tolerance creep
numeric relations get tolerances wide enough to hide real violations.
Relationships
Builds on
References
Mind
Verifying conclusions via testable predictions
Rigorous requirement definition
Nothing in this library builds on this pattern yet.