Lesson 12 ported one routine by hand, with a coding agent as the
typist. This lesson is about doing it forty more times without losing
what made the first one trustworthy: every step compared against
captured Fortran, every claim with a table attached. The rules come from
docs/AGENTS_KOKKOS_202609.md (written to be dropped in as
AGENTS.md), the plan from the proposal, and the diagrams
from its mind maps (pubs/proposal/mapas-mentais.pt.md).
Koldunov et al. (2026) took FESOM2 (about 74 thousand lines of
Fortran) to C and then to C++/Kokkos in weeks, not
years, with an LLM coding assistant directed by the model's own
experts (pubs/proposal/pt/05-justification.md (v); details
beyond that summary ⚠). The recipe:
#ifdefs disappear) and is what the Kokkos serial backend is
compared against.
How to read this figure
Takeaway. Fortran is translated into GPU-ready C++ in small steps, and each version is checked against the one before it: Kokkos serial must be bit-identical to the C, and the GPU version is compared statistically with the CPU one.
How to read. The numbered boxes are four versions of the same routine, made in that order. A solid arrow says how the next version is written; a dashed arrow points back to the version it is checked against, and its label says how strict the check is. Both Kokkos versions must also pass the tests in the group on the right: WW3's own regtests, run with the Fortran routine and with the Kokkos routine, and then the operational case. The proposal's recipe also has an L1 rung, unit tests on a synthetic JONSWAP spectrum; this repository removed it for the Kokkos code (ADR-0010).
Not shown. The tolerances
(kokkos/tools/nccmp-tol/tolerances.txt, lesson 12), and the
shortcut taken for W3SNL1, where the C stage was skipped
and Kokkos serial was compared with the Fortran fixture directly until
that test was removed (text below).
Evidence. The recipe of Koldunov et al. (2026) as
summarised in pubs/proposal/pt/05-justification.md and
applied in pubs/proposal/pt/07-methodology.md, paragraph
Interoperabilidade e testes dos kernels (v); the test group as
amended by ADR-0010
(v).
The lab collapsed the C stage for W3SNL1 (100 lines,
already CPP-preprocessed .F90): the "bit-identical to the
C" rung became "bit-identical to the Fortran fixture on every preset"
(lesson 12), and since 2026-10-14 "agrees with the Fortran in WW3's
regtests on the CPU", because the fixture tests were removed (ADR-0010).
For W3SRCE and its 239 preprocessor guards ⚠, keep the C
stage.
The six items of lesson 12's definition of done
(AGENTS_KOKKOS §1.5, §5): kernel with heritage header;
bind(C) shim plus Fortran interface; the L1 test is
withdrawn (ADR-0010: the port has no tests of its own); the WW3 regtest
replay and the regtest matrix on the CPU
(just smoke-ww3-gpu, just full-ww3-gpu,
just ww3-gpu-matrix); a PORT_STATUS.md timing
line; a property checked inside a regtest run where physics allows. Any
algorithmic change is a separate PR with its own L2 evidence (v).
The port order is the wall-clock time each routine takes in the
operational case, most expensive first. AGENTS_KOKKOS §2.2
lists effort, Kokkos suitability and ensemble payoff to plan each task,
but they do not change the order. The shares are priors until §2.4's
profiling (lesson 09) measures them and re-sorts the list (v).
Summarised:
| # | Routine(s) | Kokkos shape | Phase |
|---|---|---|---|
| 1 | W3SNL1 + INSNL1 |
team per point, tables shared; done, lesson 12 | 1 |
| 2 | W3SRCE driver |
one team per point calling inline device functions; the routine that makes or breaks device residency | 1 skeleton → 2 |
| 3 | W3SIN4, W3SDS4, W3SPR4 (or
ST6) |
inline device functions inside the W3SRCE kernel;
team_reduce for integrals |
1 |
| 4–5 | W3XYP2 + W3QCK*;
W3KTP2/3 |
MDRangePolicy<Rank<3>> per member with
halos via MPI on device buffers; team per point with 1-D sweeps over θ
then k |
2 |
| 6 | W3GATH/W3SCAT |
eliminate: one decomposition, device-resident VA |
3 |
| 7 | W3SBT*, W3SDB1, W3STR1,
W3SIC* |
trivial once #2 exists | 1–2 |
| 8–9 | output integrals (w3iogomd); forcing interpolation
(w3updtmd) |
parallel_reduce on device, copy only 2-D fields; the
forcing update is simple but triggers the H2D copy |
2 |
| 10 | restart and field I/O | not Kokkos work: async, grouped | 3–4 |
PDLIB and the exact/GMD Snl variants are
explicitly last (v). Phases (§3.6): 0 profile and ledger; 1
W3SNL1, ST4, W3SRCE skeleton with copy-in/out,
exit on L2 parity on three regtests; 2 device-resident state; 3
propagation and halos on device; 4 the member dimension; 5 flip to a C++
driver (v).
How to read this figure
Takeaway. The GPU code is added next to the original Fortran, not in place of it; a switch read at run time picks which one runs, so both are tested with the same executable.
How to read. The large box is one compiled WW3 executable. The diamond is the run-time switch; its two arrows are the two paths a call can take, the original Fortran or the new one through the C interface into the Kokkos kernel. Below the box, the same kernel runs on a CPU or a GPU backend. The dashed arrow is testing: the regression matrix and the comparator exercise both paths of the same executable.
Not shown. How state moves between Fortran and the device beyond phase 1's copy in and out per call; that is the data-residency ladder in the next section.
Evidence. W3KOKKOSMD and the
ww_kokkos_init contract, lesson 12, section The shim
and W3KOKKOSMD (v);
pubs/proposal/pt/07-methodology.md, paragraph
Interoperabilidade e testes dos kernels (v).
This is W3KOKKOSMD and WW_KOKKOS_SNL1 from
lesson 12, drawn; "measured per step" is ww_bench_snl1's
shim-versus-kernel gap.
AGENTS_KOKKOS §3.4 (v), the plan for making the phase-1
copy disappear:
| Step | On the device | Crosses per time step | Effect |
|---|---|---|---|
| 1 | nothing persistent; VA copied in and out per call |
full spectra, twice | correctness only; may be slower than Fortran |
| 2 | VA, grid metadata, tables; source terms on device |
forcing (small), output fields (small) | source-term speed-up realised |
| 3 | + propagation with device halo buffers | halos, forcing, output | transposes gone; Fortran only sequences calls |
| 4 | + member dimension | same, batched | ensemble throughput |
Where the port stands on this ladder, on the CPU only (v, PR #179,
2026-10-17): with --kokkos resident (W3SRCE,
the PR3/UQ propagation W3XYP3/W3KTP3, the
gridded output integrals of W3OUTG, and the point output
and ice update reading or zeroing only the columns they need) the
spectrum crosses to the device once and back once on
bench_small, the W3SRCE inputs only when their
bits change, and every output is byte-identical to the Fortran of the
same executable and to --ww3. That is step 2 with part of
step 3, on one process; nothing has run on a GPU, and none of these
parts is regtested yet (kokkos/ports.tsv) (WFIP-0005).
Step 2 is where the OpenACC port of Ikuyajolu et al. (2023) stalled,
because the state lived in Fortran modules; WAM6-GPU (Yuan et al. 2024)
went the other way, refactoring the whole model so the fields stay on
the device, and reported an order-of-magnitude gain on a multi-GPU node
(pubs/proposal/pt/05-justification.md (v)). The plan does
step 2 in C++ ownership (a state object created by
ww_kokkos_init, seeded by the Ctx in
snl1_shim.cpp with its persistent buffers) rather than
mirroring the modules (v).
How to read this figure
Takeaway. The project tries cheap changes before expensive ones, and a change only moves on if the forecast it produces still matches the reference within the agreed tolerances and, in step 4, if it is measurably faster.
How to read. Top to bottom. The three boxes at the top fix the reference and the measuring instruments (benchmark and profile). Each group is one step: what changes, then a diamond with its acceptance test; yes leads to the next step, and no discards the change in steps 1 to 3 and, in step 4, leads to the report of the limit. The last step ends either in operation or in a measured limit.
Not shown. The per-field tolerances, which LabECO sets for fields such as significant wave height, peak period and peak direction, and the return of a rejected change to the rewrite.
Evidence. pubs/proposal/pt/04-scope.md
(the four steps) and pubs/proposal/pt/07-methodology.md,
paragraph Critérios de concordância e ordem dos testes (v).
Lessons 09, 10 and 12 are rungs; the gates are the matrix and
nccmp-tol (the fixture tests of the Kokkos port were
removed by ADR-0010). An agent may work on any rung; it may not skip
one.
How to read this figure
Takeaway. A rewritten piece of the model is used for real forecasts only if its results agree with the original's within the agreed tolerances and it makes the whole forecast faster on the GPU.
How to read. Left to right. Each diamond is a test, taken in order; a no ends in a box that says what happens instead, and only three yes answers reach the decision with LabECO.
Not shown. The parity tolerances (lesson 12) and the timing protocol (opening paragraph of the methodology): at least five repetitions on a dedicated node, reported as median and interquartile range.
Evidence. The paragraph below the figure and
pubs/proposal/pt/07-methodology.md, paragraphs
Interoperabilidade e testes dos kernels and Riscos
(v).
Two documents, one meeting: the timing table of the operational case
with and without the kernel, and the parity report. If the H100 does not
win, the deliverable is the measured limit and a recommendation
not to operate on the GPU
(pubs/proposal/pt/07-methodology.md (v)).
Give it, every time (AGENTS_KOKKOS §1.6 (v)): the exact
routine with file path and WW3 commit; the regtest that exercises it and
the reference output; the interface contract (which arrays cross, in
which layout, which precision); and the phase, so it knows which
optimisations are off limits. The document's own example is the prompt
behind lesson 12: "Port W3SNL1 (DIA) from
model/src/w3snl1md.F90 @ WW3 7.14 … Phase 1: translate only
… Write L1 test against a JONSWAP fixture … Do not touch the Fortran
caller except to add the bind(C) interface behind the
WW_KOKKOS_SNL1 switch." Since ADR-0010 the "Write L1 test"
line is dropped: the prompt asks for the regtest replay on the CPU
instead.
Forbid explicitly (v): optimising while translating; replacing
preprocessor switches with runtime ifs inside kernels;
claiming a speed-up without an attached timing table; editing WW3
physics to "make interop easier"; introducing Unified Memory to avoid
thinking about transfers. And require it to ask before changing
a loop order that affects reduction order, a precision, a limiter or an
integration scheme (§5).
Two lab-specific lessons from the first port
(kokkos/README.md (v)): an agent reaches for
std::pow where gfortran multiplies, and for a fused
multiply-add where the Fortran rounds twice. Neither is wrong C++; both
failed the 1e-5 parity test the port had then. Run
just smoke-ww3-gpu first: the regtest is the only place
that catches them now (ADR-0010).
WAVEWATCH IV is NOAA's rewrite (plan: Office Note 525, 2025; Phase
II, languages, governance, architecture, closed March 2026, Office Note
528). As of 2026-09-15 it has a C++ core with no physics, L1 and L2
tests in GoogleTest of four planned levels, and an open CPU–GPU
architecture question with Kokkos proposed as the abstraction layer
(pubs/proposal/pt/05-justification.md (v); lesson
14). ON 525 retires WW3 only once WW4 matures, so WW3 stays
operational for years. The project's artefacts are shaped to be reusable
there (L2 replays of WW3's regtests, heritage headers; the L1 per-kernel
GoogleTests on synthetic spectra were removed by ADR-0010) without
contributing to WW4 or depending on it
(pubs/proposal/pt/04-scope.md (v)). Licence
(AGENTS_KOKKOS §4): the translated kernels derive from WW3,
so snl1_dia.cpp, snl1_tables.cpp and the
Fortran module that patches into the model, w3kokkosmd.F90,
carry LGPL-3.0-or-later; the C++ tooling around them,
snl1_shim.cpp and the C header included, is MIT (v); and
nothing here is called WW4.
pubs/proposal/refs.bib