Learn how to evaluate probabilistic forecasts and how CRPS relates to other metrics
If I asked you how to evaluate a regression problem, you would probably name quite a few evaluation metrics, such as MSE, MAE, RMSE, MAPE, etc. What these metrics have in common is that they focus on point predictions.
The situation changes a bit when we want to train our models to focus on predicting distributions instead of a single point. In that case, we need to use different metrics, which are not as commonly covered in data science blog posts.
Last time, I looked into quantile loss (a.k.a. pinball loss). This time, I will walk you through another metric used to evaluate probabilistic forecasts — the Continuous Ranked Probability Score (CRPS).
The first concept is an easy one, but it is still important to make sure we are on the same page. Probabilistic forecasts provide a distribution of possible outcomes. For example, while point forecasts would predict tomorrow’s temperature as exactly 23°C, a probabilistic model might predict a 70% chance the temperature will be between 20°C and 25°C.