Evaluation¶
Evaluation measures how well a model run reproduces observed river flow. It compares the simulated flow in a model channel with the discharge observed at a gauging station on that channel. Run reviews, calibration reviews and the public pages use its results.
Observations¶
Observed discharge comes from two sources. The scores use the GRDC observations only, for now.
- the Global Runoff Data Centre (GRDC), as monthly discharge in m³/s at stations in each region;
- contributed observations, sent as an Observations contribution (see Calibration and contributions). Once approved, they are stored as stations of the region, with the contributor as their source. They are not yet matched to channels or used for scoring.
Every station has an identifier, a name, a location, a source, units, a time step and a period.
While the licence for the GRDC observations is being settled, observed values are shown only to signed-in users. The metrics and the simulated flow are public.
Matching stations to channels¶
Each station is paired with a channel of the region's river network. The pairing comes from the first of these that applies:
- A reviewer. A pairing set by hand by the CoSWAT team is kept when the matching runs again.
- The CoSWAT framework's lookup, where the framework lists the station with its channel. Such a pairing counts as confirmed. No region has this lookup yet.
- The framework's outlet. When the framework delineated the model, it placed a subbasin outlet at the station; the station is paired with the channel that ends at that outlet. Most stations in the Save and Limpopo are paired this way.
- The platform's matching, for every other station. The nearest channel is taken. Where the station records its catchment area, the nearest channel within 5 km whose upstream area is within 25% of that area is preferred. If there is none, the nearest channel is kept and the pairing is marked to review. A station more than 5 km from every model channel is unmatched.
Stations marked to review or unmatched are not scored until someone on the team confirms or sets their pairing.
Metrics¶
For each run and each matched station, four metrics compare the simulated monthly flow S with the observed monthly flow O, over the months where both have values. A station needs at least 24 such months to be scored; stations with fewer are listed as not scored, with the reason. Metrics are monthly; daily metrics are not available yet.
| Metric | Formula | Best value | Reading |
|---|---|---|---|
| NSE, Nash–Sutcliffe efficiency | 1 − Σ(O − S)² / Σ(O − Ō)² | 1 | Below 0, the observed mean predicts better than the model. |
| KGE, Kling–Gupta efficiency | 1 − √((r − 1)² + (α − 1)² + (β − 1)²) | 1 | Combines correlation r, variability ratio α = σS/σO and bias ratio β = μS/μO. |
| PBIAS, percent bias | 100 × Σ(O − S) / ΣO | 0 | Positive when the model is too low, negative when it is too high. |
| MAE, mean absolute error | Σ|O − S| / n | 0 | In m³/s. |
Ō is the mean of the observations and n the number of months. KGE is the 2009 form (Gupta and others, 2009). NSE is not defined when the observations do not vary, and KGE when either series does not vary.
For a region, the summary is the median of each metric over its scored stations, with the number of stations matched, evaluated and scored.
Comparing runs¶
When two runs are compared (a calibrated run against the one before it, or two competing calibrations), NSE, KGE and PBIAS are shown for each shared station as old → new.
A station counts as better or worse by its KGE; a change smaller than 0.005 counts as unchanged. Until another rule is set, the recommended run is the one with the highest median KGE over the affected stations: those in the contribution's subregion or slice and everything downstream of it. A run that makes more stations worse than better is never recommended. Reviewers see the recommendation and make the decision.
Where results appear¶
On Explore, the Gauges layer shows the stations coloured by a metric you choose, with a matching legend. Clicking a station, or a river reach with a matched station (see river details), shows its simulated and observed monthly flow and its metrics.
On the Calibration page, Model performance gives the median scores of a region or slice and the scores of each gauge in a slice.