Symmetric MAPE is not symmetric

Introduction

I recently came across this interesting discussion about percentage errors and I would like to talk about MAPE and its characteristics.

MAPE is the acronym of Mean Absolute Percentage Error and is defined as

\[MAPE = \frac{100}{N} \sum_{t=1}^{N} \frac{|A_t - F_t|}{|A_t|}\]

being \(A_t\) the Actuals and \(F_t\) the Forecasts

There is some confusion and disagreement about the behaviour of MAPE.

MAPE express the deviation in relative terms and provides a simple interpretation of the error. It is easy to calculate and communicate and probably that is why it has been widely used in forecasting business.

However, it suffers from some known issues.

The problems

• MAPE ranges from 0 to \(\infty\): it diverges with \(A_t \to 0\) which leads to problems when dealing with data with zero values such as intermittent demand data or energy forecasting. This problem is even worse when working with data with arbitrary zero values, e.g, forecasting temperatures near 0\(^o\) in Celsius or Fahrenheit scale. To be fair, this would be an issue using any percentage error metric.

• MAPE is said to be asymmetric in the sense it puts heavier penalty on negative errors. When \(F_t < A_t\) the maximum possible error value is 100%, however there is no limit when forecasting on the high side. Besides, given the same absolute difference \(|A_t - F_t|\), MAPE is greater when \(F_t > A_t\):

However, there is some controversy over this last point. Some people argue this is a false dichotomy because it doesn’t make sense to compare two situations where you are exchanging forecast and actual values, and defend MAPE actually is symmetric because you can’t get a lower MAPE just by lowering your forecasts:

The…solution?

Trying to solve the alledgelly asymmetry, some alternative versions have been proposed. The more general one is:

\[ sMAPE = \frac{200}{N} \sum_{t=1}^{N}\frac{|A_t - F_t|}{|A_t| + |F_t|} \]

First thing to notice is that the range of this symmetric MAPE is \(\big[0,200 \big]\) which is somewhat antiintuitive. I can’t see myself explaining model deviations in such metric to anybody with a non-technical background. This could be solved simply dividing by 2, although that would be an aesthetic change only. On the bright side, this version of sMAPE doesn’t diverge, which brings some sanity and stability to the metric.

But the aspect that really fascinates me is that the so-called symmetric MAPE is not symmetric. In fact, sMAPE symmetrizes the asymmetric case above, and the other way around:

• Exchanging actual and forecast values does produce the same sMAPE value.

• Modifying the forecast while holding fixed actual values and absolute deviation do not produce the same sMAPE value. This maybe the most important case.

This second point is a critical issue: simply biasing the model without improving its accuracy should never produce different error values.

Taking all this into account, the proposed metric seems to be even worse than the original one. Quite surprising!

An alternative

Some others have proposed the log ratio \(ln(F_t/A_t)\) as a better alternative to MAPE. It shows, indeed, better statistical properties than others metrics:

• Applying least squares regression to this metric estimates the geometric mean whereas minimizing MAPE or sMAPE does not lead to an established measure of location.
• Given that geometric mean cannot exceed arithmetic mean, using least squares with log ratio will be more robust to outliers than OLS.
• Its values belong to a symmetric range: \((-\infty, \infty)\). However, it shows the same asymmetric behaviour than sMAPE: Exchanging actuals and forecasts holds the error (with a negative value!), but similar absolute deviations with same actual value don’t correspond with equal error values.

If interested in more information about this less-known metric, check out A better measure of relative prediction accuracy for model selection and model estimation (Chris Tofallis, 2015)

I hope this may have brought some light into this quirky behaviour of MAPE and sMAPE.

I also wrote a more general discussion about forecasting error metrics you might want to take a look at.