Blind to Anything Taken Away

If you asked me over coffee what I have been doing this week, the honest answer is: I found out that a tool I trust is blind to half of what it is supposed to catch, and that it had been hiding this from me in a fairly specific way. This is the last part of an arc whose big idea died a fortnight ago. The instruments outlived it, and they keep turning out to be more interesting than the thing they were built for.

The number I was pleased with only covered one thing

Here is the setup, in the plainest words I have.

I have a piece of music and a tool that reads it. The tool does not know what the music is supposed to sound like. It just goes through the track in five-second slices and asks, for each slice, whether anything about it stands out compared to the rest of the track. If something does, it raises a flag.

To find out whether that works, I plant faults on purpose. I take a clean track, damage it in a way I control exactly, and see whether the tool notices. Last time I planted a steady tone — think of a faint whistle sitting under the music — and turned it down step by step until the tool stopped flagging it. It stopped at about −40 dBFS, which is quiet: far below the music, but not nothing.

I was pleased with that. It turns we found nothing into a sentence with a number in it, which was the whole point of the previous part.

Then I reread what I had written underneath the result. Measured one type, one frequency, one song.

I can plant five kinds of fault, not one:

a tone — something added, a whistle
clicks — something added, tiny sharp pops
a band cut — something removed, the top of the sound sliced off
an MP3 round-trip — a bad copy, the damage a low-bitrate codec does
a mel round-trip — a smeared copy, fine detail blurred away

Only the first of those had a number. So I went to get the other four.

The bench could not do it, and would not have told me

I expected this to take twenty minutes. My notes from the day before said the other four types were “one command away”.

They weren’t, and the way they weren’t is the first thing worth telling you.

The calibration bench takes a list of strengths and plants the fault once at each. But “strength” means something different for each kind of fault. For the tone it is a volume in decibels. For a band cut it is the frequency you cut above. For MP3 it is the bitrate. The bench was writing every number I gave it into the volume setting — and only the tone reads that setting. The other four just ignored it.

So I checked before running anything. I planted a band cut at “−35” and again at “−45”, and compared the two files.

They were identical. Byte for byte, the same file.

The bench would have run happily, printed six neat rows, and every row would have been the same track measured six times. It would have looked exactly like a calibration.

I fixed it so each fault type names the setting its strength actually lives in, and so the bench refuses to run if you hand it that setting yourself — the exact mistake cannot come back quietly. Then I re-ran the old tone calibration to check I had not broken anything on the way. It came back identical to the published table, to the second decimal.

Only then did I measure the other four.

What came back, and it is not close

fault	I varied	across	the tool, on its own	the checker, holding the answer key
band cut	cut frequency	16 kHz → 2 kHz	never flags it	14.09 → 21.05 dB
MP3 copy	bitrate	192 → 32 kbps	never flags it	9.03 → 43.30 dB
smeared copy	detail kept	256 → 24 bands	only at the very worst	5.38 → 10.01 dB
tone	volume	−35 → −50 dBFS	down to −40	14.03 → 2.46 dB
clicks	loudness	0.8 → 0.1	down to 0.3	10.98 → −0.00 dB

The two it catches are the tone and the clicks. Both of those add something to the track — something narrow, something sharp.

The three it misses all remove or smear.

The band cut is the one that stopped me. At the bottom of that row I am chopping everything above 2 kHz off a six-and-a-half second stretch of music. That is not subtle. Half the spectrum is gone; anyone would hear it instantly as the music going muffled and then clearing up again. The checker that holds the answer key measures it at 21 dB, which is enormous.

The tool never says a word. Not at 2 kHz, not anywhere in between, not once.

Same for the MP3 copy at 32 kbps, which sounds like a bad phone call.

Why: the ruler is made out of the track

This is the part I actually enjoyed.

The tool has no idea what music should sound like, and that is deliberate. It gets its sense of normal from the track itself: it looks at all twenty-one slices, works out what a typical slice looks like and how much slices normally vary, and flags anything sitting too far outside that. It is a lovely property, because it works on any material with no training and no reference recording.

But look at what happens when I plant something.

The fault joins the comparison group.

Take the worst band cut. In the slice where I planted it, the energy up in the treble drops by 8.26 dB. If I measure that against a clean copy of the same track, it lands at 6.3 standard deviations out — vastly past the line the tool draws at 4. It should be flagged and it should not be close.

But the tool is not comparing to a clean copy. It is comparing to this track, and this track now contains my fault. And the fault does not just sit low; it stretches the range of normal the track has:

treble energy across the track	typical slice	how much slices vary
clean	−45.77	1.19
with the band cut planted	−46.54	2.15

The variation nearly doubles, and that number is the denominator for every slice. So my fault, which was 6.3 deviations out, becomes 3.1 deviations out — under the line, unflagged.

And the same widened ruler is applied to everything else. There was a genuinely odd slice at the very start of this track, 6.9 deviations out, which the tool had always flagged. With my fault planted elsewhere it drops to 3.5, and the flag disappears.

MATH BOX

Masking and swamping, and why deleting one point does not help

This has names. In outlier detection, masking is when an outlier inflates the estimates of centre and spread enough that it — or another outlier — falls back inside the acceptance region and escapes detection. Swamping is the mirror image: a perfectly ordinary observation is flagged because the outlier has dragged the estimates. Both are standard failure modes of any rule that estimates its own reference from contaminated data, and both are in the classic treatments (Barnett & Lewis, 1994; Hadi & Simonoff, 1993; Davies & Gather, JASA 1993).

The reason they are not symmetric here is the breakdown point. The median tolerates contamination up to half the sample, so the track’s typical value barely moved (−45.77 → −46.54, on a scale where the effect is 8 dB). The scale estimate is where the damage lands. The rule fires at k · MAD with k = 4, and the MAD went 1.19 → 2.15, so the fault raises the bar it then has to clear — the same mechanism as the flat-artefact result in the previous part (Hampel, 1971; Rousseeuw & Croux, JASA 1993).

And the obvious repair fails for a documented reason. The natural fix is to judge each slice against a reference computed with that slice left out, so a fault cannot prop itself up. That is single-deletion diagnostics, and its classic weakness is precisely multiple outliers: with two or more contaminated points, deleting one leaves the others still inflating the estimate. It is why the field moved to high-breakdown estimators computed from a clean subset rather than one-at-a-time deletion (Rousseeuw & Leroy, 1987). My planted fault spans two slices of the grid. One deletion was never going to be enough, and the measurement below says so in numbers.

The detection floor itself is the older idea: a limit of detection is a property of method and matrix together, not of the instrument alone (Currie, 1968; IUPAC). “−40 dBFS” was never a property of my tool. It was a property of my tool, that music, and that one fault type.

The same fault, the same measurement, two different reference distributions. On the left it is unmissable. On the right it has widened the ruler it is being measured with — and pushed one of the track's own findings off the edge.

It also goes wrong in the wrong place

Two things fall out of that mechanism, and both of them land away from where the fault actually is.

It raises alarms somewhere else. When I plant the tone at −35 dBFS between 61.4 and 68.2 seconds, the tool flags a slice at 5 to 10 seconds — fifty seconds away, where I planted nothing at all. The tone lifted the noise floor in its own slice, that shifted the reference for the whole series, and a quiet slice at the other end of the track crossed the line as a result.

And it deletes findings somewhere else. The band cut and the MP3 copy each take the track’s own flag count from 20 down to 16 — and not one of the four that vanished is anywhere near where I planted the damage.

I had seen a version of this before, but only with a fault smeared across the entire track, which felt like a special case. It is not. Six and a half seconds is enough.

I wrote down the fix, tested it, and it lost

I thought the repair was obvious, and I want to be honest that I believed it for about ten minutes.

If the problem is that the fault gets into its own reference group, then leave it out. Judge every slice against a reference built from all the other slices. Simple, cheap, one line.

I ran the null test first — what does this rule do to a clean track, where every flag it raises is a false one — because the whole point of the previous part was that you measure the empty case before you interpret anything.

rule	flags raised on the clean track	finds the band cut?	finds the MP3 copy?
current	20	no, at any setting	no, at any setting
leave-one-out	24	no, at any setting	no, at any setting

More false alarms, not one new detection.

The reason is in the MATH BOX and it is not subtle once you see it: my planted fault covers two slices of the grid. Leave one out and the other one is still there, still stretching the ruler. To leave both out I would have to know where the fault is, and if I knew that I would not need the detector.

So there is no cheap fix. The rule would need a reference from outside the track — a clean copy to compare against. Which is exactly what the checker holding the answer key has, and exactly why it sees all five faults at every strength while the tool on its own sees two.

That is a tidy little circle to end an arc on. The thing that separates a detector from a checker is not the cleverness of the rule. It is whether it has something honest to compare to.

NERD BOX

A third disagreement, inside a single function call

The click meter has its own version of this problem and I only found it because a column of results looked too tidy.

It reports the sharpest transient in the window as a ratio against the sharpest transient in the clean reference, and it counts how many clicks it found using a threshold 6 dB above anything the clean window produced. Two readings, one call.

At click loudness 0.4 and 0.3 the meter reports the fault as present — 5.29 dB and 3.01 dB above the music — while the counter reports zero clicks found. Both numbers come out of the same function on the same audio. They are not wrong, they are answering different questions with the same word, and a report that quotes only one of them sounds certain either way.

I also spent a minute convinced the meter was broken, because below that it reads exactly −0.00 at every level instead of decaying smoothly. It is not broken. It is a ratio of maxima, so once my click sits under the music’s own sharpest transient the maximum is the music’s own transient, unchanged, and the ratio is exactly one. The meter has a hard floor rather than a gentle fade — which is worth knowing before you read a small number as “almost detected”.

What I would keep from this

A detector that builds its reference out of the thing it is inspecting can be defeated by the thing it is inspecting. Not fooled by a clever attacker — defeated by an ordinary fault, arithmetically, with nobody trying.
Ask which direction your detector works in. Mine notices additions and is blind to subtractions, and nothing in its description says so. I had used it for weeks without asking.
A single flag is not a location. Mine raises alarms fifty seconds from the damage. If I had shipped a tool that pointed at a timestamp, it would have been pointing at innocent audio.
Fewer warnings is not a cleaner track. This is the one I would put in the specification in bold.
A detection floor belongs to a fault type, not to a tool. “−40 dBFS” was true and I was quoting it too broadly by exactly four fifths.
An instrument that degrades silently is worse than one that fails. Mine would have printed a perfectly formatted calibration of the same file six times. The check that caught it took two minutes and I nearly skipped it.

The concept that all this was built for died two weeks ago, on a kill condition I had written down in advance. The instruments have now overturned three of my own written-down conclusions since — which is a strange thing to be pleased about, and I am.

Sources & further reading

V. Barnett & T. Lewis, Outliers in Statistical Data (3rd ed., 1994) — masking and swamping as the two standard failure modes of self-referential outlier rules.
A.S. Hadi & J.S. Simonoff, Procedures for the Identification of Multiple Outliers in Linear Models (JASA, 1993) — why one-at-a-time deletion is defeated by more than one outlier.
P.L. Davies & U. Gather, The Identification of Multiple Outliers (JASA, 1993) — outlier identifiers, and the price of estimating your own reference from contaminated data.
P.J. Rousseeuw & A.M. Leroy, Robust Regression and Outlier Detection (1987) — the high-breakdown alternative to single-deletion diagnostics.
F.R. Hampel, A General Qualitative Definition of Robustness (Annals of Mathematical Statistics, 1971) — the breakdown point, and why the median held while the spread estimate did not.
P.J. Rousseeuw & C. Croux, Alternatives to the Median Absolute Deviation (JASA, 1993) — what the MAD is and is not immune to.
L.A. Currie, Limits for Qualitative Detection and Quantitative Determination (Analytical Chemistry, 1968) — the detection limit as a property of method and material together, not of the instrument alone.

— SF3D