Data integrity: detecting losses, drifts and truncated recordings

The file exists. So what?

The end of a sortie produces a file of the expected size that opens without complaint and draws curves that look like the aircraft. Every one of those facts is compatible with a recording that has lost a third of a channel, drifted half a second against another unit, or captured a folded frequency that was never there.

The good news is that a properly built container carries the evidence needed to prove it: a synchronisation pattern on every block, a per-channel sequence counter, lengths, checksums and a time stamp. The anomalies are detectable. What is usually missing is anybody looking.

The taxonomy — and what each one costs you

Read the last column first. Six of these eight announce themselves to anyone who checks. The last two do not, and they are the two that invalidate conclusions rather than lose data — which is why they belong to the pre-flight discipline and not to the post-flight report.

Sync losses, and what they really mean

On a telemetered stream, a sync loss is a link event: the receiver lost frame lock and everything until re-lock is gone. It is normal, it happens on every campaign, and the useful information is the pattern — how often, how long, and at which point of the flight. A cluster of losses in every high-bank turn is an antenna problem, not a data problem.

Inside a recorded file, a sync loss means something different and more serious: the writer produced something the reader cannot follow. That points at the equipment or the medium rather than the radio path, and it deserves investigation rather than a note in the log.

Sequence breaks and missing blocks

This is the anomaly that hides best. Each channel carries its own block counter, incremented on every block including the empty ones. If the counter jumps, blocks are missing — and because each remaining block still carries its own time, the file reads perfectly: the curve simply steps over the gap, and if the gap is short nobody sees it.

The causes are usually mundane and worth knowing: a buffer that overran because the medium paused, a write that arrived late, a network packet dropped between a slave unit and the master. All of them are capacity or timing problems, all of them get worse as the medium fills, and all of them are invisible without checking the counter.

Checksums, and corruption that keeps quiet

A checksum is the only mechanism that catches data which is present, well-formed and wrong. Without one, a bit flipped on the medium or in transit becomes a value — a plausible one, in the right place, at the right time.

Which is why it matters whether your container actually carries and verifies one, and whether your reader checks it or skips it for speed. A reader that silently ignores a failed checksum is worse than one that has none, because it produces confidence.

Time drift, and why it is the expensive one

A time anomaly does not damage a channel — it damages the relationship between channels, which is where the engineering lives. Two units that were never locked to a common reference produce two internally consistent datasets separated by an offset that appears in neither file.

Drift is the same problem in slow motion: a unit in holdover after losing GPS walks away at its oscillator specification, a few milliseconds an hour on an ordinary part. Over a three-hour sortie that is enough to ruin a phase measurement and not enough for anyone to notice.

The detection is possible but only if you prepared for it: record the time source and its health with the data, and inject a common event on every unit before the flight so there is a measured reference to check against. Without one of those two, a time offset is undetectable and permanent.

Two checks, before and after

Integrity is not an analysis activity, it is an operations activity, and it belongs in two short procedures rather than in somebody good intentions.

Before the flight: the setup record matches the intended parameter list; every configured channel produces data on a bench run; the anti-aliasing filters match the rates actually assigned; a common event is injected on all units and the offsets are measured; the medium has room for the sortie plus margin.

After the flight, before the aircraft flies again: the file opens and closes cleanly; no channel is missing against the setup record; sync losses and sequence breaks are counted and located; checksums pass; the saturation counters are read; and the recording is downloaded rather than left to be overwritten.

The second procedure matters more than it looks, because its purpose is not to validate the data. It is to catch an installation fault while the aircraft is still on the ground, so the next sortie does not repeat it.

What an integrity report should contain

• One line per channel, not one verdict per file. A recording is rarely uniformly good or bad.
• Anomalies classified by type, because the type is what tells you where to look — a link, a buffer, a medium, a configuration.
• The position and duration of each one, so an analyst can decide whether it touches the segment they care about.
• A comparison against the setup record, so a channel that is absent is reported rather than assumed.
• And a stable format, so two reports from two campaigns can be compared and a trend becomes visible before it becomes a failure.

The reason to automate this is not thoroughness, it is honesty. A check that a human performs when they have time is a check that is skipped exactly when the campaign is busy — which is exactly when the anomalies happen.