Benchmark
WAV suites

Download

Every file 16 bit PCM, mono, 12000 Hz — the format WSJT‑X and JTDX read natively. Each suite carries SHA256SUMS. Full description: README.md.

Run it

You need Python 3.6 or newer and the standalone decoder jtdxjt9, which ships in every JTDX Contest Edition download — C:\Program Files\JTDX_contest\bin\jtdxjt9.exe on Windows, or squashfs-root/usr/bin/jtdxjt9 after running the AppImage once with --appimage-extract. Unpack a suite so its directory sits beside bench/:

Linux, macOS                          Windows
./bench/run_ft8.sh --periods 40       bench\run_ft8.cmd --periods 40     a first look, ~5 min
./bench/run_ft8.sh                    bench\run_ft8.cmd                  the whole hour
./bench/run_ft4.sh --presets all      bench\run_ft4.cmd --presets all    every preset
./bench/run_ft8.sh --crowded          bench\run_ft8.cmd --crowded        scored true / false
./bench/run_ft4.sh --list             bench\run_ft4.cmd --list           what the presets are

It prints a table per run and, if you pass --json, the same as JSON. Nothing is written outside a scratch directory that is removed again; the WAV files are only read.

What comes out

preset          decodes RX s mean RX s max bg s mean total s/period  wall s  in 2.7s
------------------------------------------------------------------------------------
maxeff             5417      0.40     0.73         -           0.40      97  240/240
light              6091      1.13     1.74      4.20           5.33    1275  240/240

decodes are distinct messages summed over the periods. RX s is the phase that has to decide your reply; bg s is the background phase that runs on while you transmit. The last column counts the periods whose RX phase finished inside the mode's reply deadline — 2.7 s for FT8, 1.36 s for FT4 — and it is the column that decides which preset a given machine can actually run. A second table puts the same run beside the reference machine and ends with a speed index.

The reference numbers

AMD Ryzen 7 5800H, 8 cores / 16 threads, powersave governor · Linux Mint 22.3, gfortran 13.3 ‑O3 · 12 decoder threads · engine jtdxjt9 8a3e17ee (the 3.0.0‑rc02 decoder), 2026‑09‑05 · mycall CE3TSK FF46, 100–3100 Hz, all 240 periods. total s is RX + background per period.

FT8, the one-hour suite

The three pipeline presets need the two-period background window — the 15/15 TX cycle with the background enabled — which is why their total is past 15 s while their RX phase is not.

FT4, the one-hour suite

The stock and WSJT-X rows are there for scale. The script does not run them — they need those programs' own decoders — only the JTDX Contest Edition presets.

Reproducing them

The decode counts depend on more than the audio. To land on the numbers above, keep --mycall CE3TSK --mygrid FF46 (a different call changes what the AP search and the hint memory look for), --low 100 --high 3100, --threads 12, and the whole 240 periods in one run — the hint memory carries from period to period inside one process, as it does on air. A different build will not reproduce them to the message: expect a drift of a few messages in a few thousand between releases.

What the recordings are

Raw off-air audio of a real band: real callsigns, real QSOs, the noise, the splatter and the stations that never quite decode. Nothing was selected, trimmed or cleaned, and the periods are consecutive — a benchmark on clean synthetic signals answers a different question. There is no ground truth for them, so what is measured is how many distinct messages a preset recovers and how long each period took; the honest comparison is between presets and between machines on the same files.

The crowded band, scored

--crowded scores over the band the manifest's own signals occupy, not the on-air 100–3100 Hz: the FT4 twin reaches 4922 Hz, and 40 of its 103 signals sit above 3100 Hz, so the on-air range would mark a preset down for signals it was never allowed to look at. --low and --high override it. On the reference machine, with this release's engine:

found is the first, cold period. best of 5 is the best single period when the same file is handed to the decoder five times running — --crowded --repeat 5, which is what re-opening it in the program does. The audio is identical in all five, so the difference is purely what the hint memory carried forward: +1 to +7 on FT8, 0 to +2 on FT4. Note that it is the shallow presets that gain most — the deep ones had already found nearly everything on the first pass — and that nohint, which switches the memory off, gains exactly nothing, which is the control behaving.

Not one message outside the manifest, at any depth, in either mode, cold or repeated — which is the point of having a truth set at all.

The crowded-band files are the exception. ft8_full_band_16.wav carries 104 FT8 signals across the band; it is the recording 210615_071015.wav — 2021‑06‑15 at 07:10:15 UTC — a band packed with Japanese stations: 101 of its 104 senders are JA, answering Cyprus, Israel, European Russia, Crete, Ukraine and Mongolia. Every message in it is corroborated across several decoder families.

ft4_full_band_16.wav is a synthetic FT4 version of that same period: the same message set at the same SNRs and DTs, with the frequencies stretched by the FT4/FT8 signal-width ratio so every overlap is the same overlap in signal widths. One message would have landed above the band and was dropped, which is why it holds 103 and not 104. Because that file was built from its manifest, anything else printed from it is demonstrably a false decode — which is what --crowded counts.

Expect FT4 to find fewer of them, and not because of the decoder: FT4 is intrinsically less sensitive than FT8 — its transmission is about a third as long, so its decoding threshold sits several dB higher — and the weakest signals of the set fall below it. The deepest FT8 presets reach 101 of 104 here, the deepest FT4 presets 77 of 103. That gap is the difference between the two modes on identical content, not between two decoders.

☕Support this work on Ko-fi