# JTDX Contest Edition — decoder benchmark suites
Three sets of recorded and synthesised WAV files, the ones the published FT8 and FT4 decoder
measurements were made on, with a script that replays them through the standalone decoder and
prints what it found and what it cost.
Two things you can do with them:
* **check the claims** — the +27.5 % FT8 and +24 % FT4 figures come from these files, on these
presets; the same script that produced them is here;
* **benchmark a machine** — the same hour on your own hardware, against the reference numbers
below. A decoder recipe is only usable if it finishes inside the mode's reply window, and
that depends entirely on the machine, so this is the number that decides which preset you
can actually run.
Everything is free software under GPL v3, as the program is.
---
## The suites
| Suite | Files | Size | What it is |
|---|---|---|---|
| `ft8_onair_1h` | 240 | 86 MB | One hour of 14.074 MHz FT8, 16:57–17:57 UTC 2026‑08‑27, recorded off the air in Santiago, Chile — every 15 s period, none skipped |
| `ft4_onair_1h` | 240 | 35 MB | 240 FT4 periods on 14.080 MHz, 23:38 UTC 2026‑08‑29 to 00:38 UTC 2026‑08‑30, during WW Digi weekend |
| `crowded_band` | 2 + manifests | 0.5 MB | One deliberately crowded synthetic period per mode, every signal in it known — the only files here with a truth set |
Each suite is both a directory of individual files and a `.zip` of the same, with `SHA256SUMS`
inside. Take the zip unless you only want one period.
All files are **16 bit signed PCM, mono, 12000 Hz**, the format WSJT‑X and JTDX use natively —
so they open in any of them, in Audacity, or in `File → Open` in the program itself. The
captures were made when this fork carried 32 bit samples; they were contracted with the same
arithmetic shift the program's own reader uses, so a file here decodes exactly as the 32 bit
original did.
### About the on-air suites
They are raw off-air audio of a real band: real callsigns, real QSOs, everything that was
audible at that receiver in that hour, including the noise, the splatter and the stations that
never quite decode. That is the point — a benchmark on clean synthetic signals answers a
different question. Nothing was selected, trimmed or cleaned; the periods are consecutive.
There is no ground truth for them. What is measured is **how many distinct messages a preset
recovers from the hour** and **how long each period took**, and the honest comparison is
between presets and between machines on the same files, not against an absolute.
### About the crowded-band suite
`ft8_full_band_16.wav` is one FT8 period carrying 104 signals across the band. It is the
recording **`210615_071015.wav`** — 2021‑06‑15 at 07:10:15 UTC — a band packed with Japanese
stations: 101 of its 104 senders are JA, answering Cyprus, Israel, European Russia, Crete,
Ukraine and Mongolia. `ft8_full_band_truth.tsv` lists every message, each corroborated across
several decoder families (103 confirmed, 1 plausible); a message outside that set is unconfirmed
rather than proven false.
`ft4_full_band_16.wav` is a **synthetic FT4 version of that same period**: the same message set at
the same SNRs and DTs, with the frequencies stretched by the FT4/FT8 signal-width ratio so every
overlap is the same overlap in signal widths. One message would have landed above the band and was
dropped, which is why it holds 103 and not 104. `ft4_full_band_truth.tsv` is its manifest, and
because the file was built from the manifest, anything else printed from it is demonstrably a
**false decode**.
Expect FT4 to find fewer of them, and not because of the decoder: FT4 is intrinsically less
sensitive than FT8 — its transmission is about a third as long, so its decoding threshold sits
several dB higher — and the weakest signals of the set fall below it. The deepest FT8 presets
reach 101 of 104 here, the deepest FT4 presets 77 of 103. That gap is the difference between the
two modes on identical content, not between two decoders.
`--crowded` scores over the band the manifest's own signals occupy, not the on-air 100–3100 Hz:
the FT4 twin reaches 4922 Hz, and 40 of its 103 signals sit above 3100 Hz, so the on-air range
would mark a preset down for signals it was never allowed to look at. Pass `--low` / `--high`
to override.
On the reference machine, with the engine of this release:
| preset | FT8: found of 104 | best of 5 | not in set | | preset | FT4: found of 103 | best of 5 | false |
|---|---|---|---|---|---|---|---|---|
| classical | 77 | 84 | 0 | | nohint | 76 | 76 | 0 |
| maxeff | 77 | 83 | 0 | | default | 76 | 76 | 0 |
| maxdec | 88 | 92 | 0 | | bestpower | 76 | 76 | 0 |
| light | 95 | 98 | 0 | | recommended | 77 | 79 | 0 |
| ensemble | 98 | 100 | 0 | | mostreply | 77 | 77 | 0 |
| pipeens | 101 | 102 | 0 | | maxeffort | 77 | 77 | 0 |
| pipefull | 101 | 102 | 0 | | | | | |
| piperun | 98 | 101 | 0 | | | | | |
**found** is the first, cold period. **best of 5** is the best single period when the same file is
handed to the decoder five times running — `--crowded --repeat 5`, which is what re-opening it in
the program does. The audio is identical in all five, so the difference is purely what the hint
memory carried forward: +1 to +7 on FT8, 0 to +2 on FT4. Note that it is the *shallow* presets that
gain most — the deep ones had already found nearly everything on the first pass — and that
`nohint`, which switches the memory off, gains exactly nothing, which is the control behaving.
Not one message outside the manifest, at any depth, in either mode, cold or repeated — which is the
point of having a truth set at all. The FT4 column agrees with the project's own in-tree benchmark.
**Side note — you do not need the script for this.** Every file here is an ordinary WAV, so any
of these programs opens it with **File → Open**. The crowded-band results above can be checked by
hand in a minute: put the program in FT8 mode and open `ft8_full_band_16.wav`, or in FT4 mode and open
`ft4_full_band_16.wav`, from . Every JTDX and
WSJT‑X version reports how many decodes it got, so two programs can be compared on the same period
with nothing but the file.
**Side note — opening the same file again is the next period.** The decoder treats each open as a
new period, so re-opening one file two or three times shows the hint memory working: what the first
pass learned is carried into the next, and the count rises. On `ft8_full_band_16.wav` the third open
decodes between two and six more than the first, depending on the preset — 95 then 95 then 98 on
*light*, 71 / 71 / 73 at the plain default. Nothing about the audio changed; only what the decoder
already knew.
**Side note — widening the decode range finds more, on these files only.** The FT4 twin's signals
were stretched as high as 4922 Hz, so what you find in it depends on the range you hand the
decoder: over the on-air 100–3100 Hz the default preset reaches 36 of the 103 signals, over the
manifest's 100–4951 Hz it reaches 76 — for roughly a tenth more decode time. That is a property of
the test file, not advice for the air. On a real band a wider range buys nothing: the radio's own
passband already bounds where a signal can be, and every extra kilohertz is spectrum the decoder
searches for the whole period. Which is why `--crowded` takes the range from the manifest, while
the two hour suites stay at 100–3100 Hz.
---
## Running it
You need Python 3.6 or newer and the standalone decoder, `jtdxjt9` (`jtdxjt9.exe` on Windows).
It ships inside every JTDX Contest Edition download:
* **Windows** — `C:\Program Files\JTDX_contest\bin\jtdxjt9.exe`
* **Linux, AppImage** — run it once with `--appimage-extract`; the decoder is
`squashfs-root/usr/bin/jtdxjt9`
* **built from source** — `build/jtdxjt9`
The script finds it in the usual places; otherwise point at it with `--jt9`.
Unpack a suite so its directory sits beside `bench/`, then:
```
Linux / macOS Windows
one mode, quick ./bench/run_ft8.sh --periods 40 bench\run_ft8.cmd --periods 40
one mode, full ./bench/run_ft8.sh bench\run_ft8.cmd
every preset ./bench/run_ft4.sh --presets all bench\run_ft4.cmd --presets all
crowded band ./bench/run_ft8.sh --crowded bench\run_ft8.cmd --crowded
list the presets ./bench/run_ft4.sh --list bench\run_ft4.cmd --list
```
or call the one script directly, which is the same thing:
```
python3 bench/jtdxbench.py --mode ft8 --jt9 /path/to/jtdxjt9 --json mine.json
```
`--help` lists every option. Nothing is written outside a scratch directory that is removed
again, and the WAV files are only ever read.
**How long it takes.** On the reference machine, the whole FT8 suite is 1.6 minutes for the
cheapest preset and 56 minutes for the deepest; the whole FT4 suite is 24 seconds to 9 minutes.
The default preset selection (`--presets main`) is about half an hour for FT8 and 20 minutes for FT4. Use
`--periods 40` for a first look — the timings are then already meaningful, the decode counts
are not comparable with the reference.
**Close the program first.** The seconds are only true on an otherwise idle machine.
---
## Reading the output
```
preset decodes RX s mean RX s max bg s mean total s/period wall s in 2.7s
------------------------------------------------------------------------------------
light 6091 1.13 1.74 4.20 5.33 1275 240/240
```
| Column | Meaning |
|---|---|
| decodes | distinct messages, summed over the periods (a message repeated in the same period counts once) |
| RX s | the RX phase, the one that has to decide your reply — mean and worst period |
| bg s | the background phase, which runs on while you transmit; `-` if the preset has none |
| total s/period | RX + background: what one period costs the machine altogether |
| wall s | the whole run, start to finish |
| in 2.7 s | periods whose RX phase finished inside the reply deadline — **2.7 s for FT8, 1.36 s for FT4** |
That last column is the one to look at when choosing a preset for a machine. A preset that
misses the deadline still decodes everything it would have; it just decides the reply after the
answer was due. The second table compares the same run against the reference machine and ends
with a speed index — above 1.00× is faster than the reference.
---
## The reference numbers
Reference machine: **AMD Ryzen 7 5800H**, 8 cores / 16 threads, powersave governor, 14 GB,
Linux Mint 22.3, gfortran 13.3 ‑O3, **12 decoder threads**. Engine `jtdxjt9 8a3e17ee`
(the 3.0.0‑rc02 decoder), run 2026‑09‑05, mycall `CE3TSK`, grid `FF46`, 100–3100 Hz, over the
whole 240 periods of each suite. `total s` is RX + background per period; the whole-run wall
time is that times 240.
### FT8, the one-hour suite
| preset | decodes | vs stock | RX s mean (max) | bg s mean | total s |
|---|---|---|---|---|---|
| *stock JTDX v2.2.159, 12 threads* | *4869* | *baseline* | *1.40* | *–* | *1.40* |
| *WSJT‑X 3.0.2 MT decoder, 12 threads* | *5171* | *+6.2 %* | *1.19* | *–* | *1.19* |
| classical | 5249 | +7.8 % | 0.76 (1.32) | – | 0.76 |
| maxeff | 5417 | +11.3 % | 0.40 (0.73) | – | 0.40 |
| maxdec | 5678 | +16.6 % | 1.14 (2.03) | – | 1.14 |
| light | 6091 | +25.1 % | 1.13 (1.74) | 4.2 | 5.33 |
| ensemble | 5992 | +23.1 % | 4.52 (8.01) | – | 4.52 |
| pipeens | 6194 | +27.2 % | 1.29 (2.04) | 11.4 | 12.71 |
| pipefull | 6208 | +27.5 % | 1.29 (2.07) | 12.7 | 14.03 |
| piperun | 6234 | +28.0 % | 0.40 (0.71) | 12.9 | 13.29 |
The three pipeline presets need the two-period background window — the 15/15 TX cycle with the
background enabled — which is why their total is far past 15 s while their RX phase is not.
### FT4, the one-hour suite
| preset | decodes | vs stock | RX s mean (max) | bg s mean | total s |
|---|---|---|---|---|---|
| *stock JTDX v2.2.159* | *1525* | *baseline* | *1.40* | *–* | *1.40* |
| *WSJT‑X 3.0.2* | *1561* | *+2.4 %* | *0.14* | *–* | *0.14* |
| *WSJT‑X improved 3.2.0* | *1688* | *+10.7 %* | *0.22* | *–* | *0.22* |
| nohint | 1552 | +1.8 % | 0.09 (0.18) | – | 0.09 |
| default | 1737 | +13.9 % | 0.10 (0.67) | – | 0.10 |
| bestpower | 1822 | +19.5 % | 0.10 (0.19) | 0.3 | 0.38 |
| recommended | 1881 | +23.3 % | 0.10 (0.19) | 2.1 | 2.22 |
| mostreply | 1878 | +23.1 % | 0.48 (1.08) | 0.5 | 0.98 |
| maxeffort | 1888 | +23.8 % | 0.59 (1.24) | 0.6 | 1.21 |
The stock and WSJT‑X rows are there for scale; the script does not run them, since they need
those programs' own decoders. It runs the JTDX Contest Edition presets only.
### Reproducing them
The decode counts depend on more than the audio. To land on the reference numbers, keep:
`--mycall CE3TSK --mygrid FF46` (a different call changes what the AP search and the hint
memory look for), `--low 100 --high 3100`, `--threads 12`, and the whole 240 periods in one
run — the hint memory carries from period to period inside one process, as it does on air.
Change any of them deliberately, not by accident.
A different build will not reproduce them to the message: the numbers above are one engine on
one day. Expect a drift of a few messages in a few thousand between releases; if you see more,
that is worth reporting.
---
## Files
```
ft8_onair_1h/ 240 wav + SHA256SUMS ft8_onair_1h.zip
ft4_onair_1h/ 240 wav + SHA256SUMS ft4_onair_1h.zip
crowded_band/ 2 wav, 2 truth tsv crowded_band.zip
bench/
jtdxbench.py the benchmark, one file, no dependencies beyond Python 3.6
run_ft8.sh run_ft8.cmd FT8, Linux/macOS and Windows
run_ft4.sh run_ft4.cmd FT4
README.md the short version of this page
```
## Credits and licence
Recordings and code by **Tihomir Sokcevic, CE3TSK**, released under the GNU General Public
License v3, the licence of JTDX Contest Edition — itself a derivative work of JTDX by UA3DJY
and ES1JA and of WSJT‑X by K1JT and the WSJT development group.
The recordings contain the callsigns of stations that were transmitting on an amateur band at
that time; that is public transmission, and nothing else about them is recorded here.
·