Skip to content

OOM Accuracy — Order-of-Magnitude Accuracy ×{k}

\[ \text{OOM\_Acc}(k) = \frac{1}{t} \sum_{i \in M} \mathbf{1}\!\left[\left|\log_{10} \frac{y_{\text{pred},i}}{y_{\text{true},i}}\right| \leq \log_{10} k\right] \]

\(T\) is the set of Samples with a valid target (\(y_{\text{true},i}\) finite and \(> 0\)), \(t = |T|\), and \(M \subseteq T\) the Samples whose prediction is valid too (\(y_{\text{pred},i}\) finite and \(> 0\)). The fraction is of all valid targets: a prediction that is negative, zero or not finite is within no factor and counts as a miss. A model that predicts negative values on four of five Samples therefore scores at most 0.2, not 1.0. Samples whose target is outside the domain drop out — that is the data, not the model.

COREMat reports this metric once per factor in evaluation.oom_factors, keyed oom_acc_x{k}. The defaults are ×2, ×5 and ×10; ×3.16 (≈ half an order of magnitude) is a common addition when the target spans many decades.

Implementation: custom — not from sklearn.

Running example (k = 2, a default factor):

  1. Log-distances (reusing OME calculation): \(|\log_{10}(y_{\text{pred},i} / y_{\text{true},i})| \approx [0.079, 0.046, 0.041, 0.046, 0.041]\)
  2. Threshold: \(\log_{10}(2) \approx 0.301\)
  3. Count within threshold: all 5 values \(\leq 0.301\)
  4. Result: \(\text{OOM\_Acc}(2) = 5 / 5 = 1.00\) (100%)

Interpretation

OOM Accuracy at factor \(k\) is the fraction of predictions within a factor of \(k\) of the true value in either direction. A prediction is within ×3.16 of the truth if \(y_{\text{true}} / 3.16 \leq y_{\text{pred}} \leq y_{\text{true}} \times 3.16\).

  • oom_acc_x2 = 0.90 → 90 % of predictions are within a factor of two of the truth.
  • oom_acc_x10 = 1.00 → all predictions are within one order of magnitude.

When to use

  • Targets spanning many orders of magnitude where absolute or percentage error thresholds are impractical to define uniformly.
  • When reporting model quality to domain experts who think in "factor of 2" or "order of magnitude" terms.
  • Alongside OME to give both an average log-error and a pass/fail fraction.

Disadvantages

  • Only defined for strictly positive targets; Samples with a non-positive target are dropped. A non-positive prediction counts as a miss.
  • The threshold \(k\) is arbitrary; results only become comparable across papers/models when the same \(k\) values are used.
  • Provides no information about the distribution of errors within the passing fraction.

Configuring factors

Set evaluation.oom_factors in your run config (the packaged defaults live in src/coremat/core/config/defaults.yaml) to customise which thresholds are reported:

evaluation:
  oom_factors: [2.0, 3.16, 10.0]

Compute cost

Low — single pass per factor, requires logarithm computation.

← Back to Metrics index