Using Monte Carlo to quantify the model prediction error

Monte Carlo simulations demonstrated

Prediction is very difficult, especially if it’s about the future. — Niels Bohr

As a data scientist working in the engineering field, I frequently collaborate with other research engineers on the physics side, with the task of developing quantitative models to predict physical phenomena of interest. However, getting a reliable prediction is never easy, mostly it’s because I can rarely be sure about the correct values of the model hyperparameters. Those hyperparameters, which convey specific physical interpretations, are usually calibrated via noisy experiments. As a result, there is no way to be sure of their exact values due to the randomness inherent in the calibration process. Simply put, those hyperparameters are uncertain.

I am usually left with two options: one, I can just pick the nominal values (or most probable values) of the hyperparameters and make predictions based on them, hoping that the obtained results are accurate or at least close to the truth; second, I can accept the fact that those hyperparameters are uncertain, and use proper uncertainty propagation techniques to propagate the uncertainties from model hyperparameters to the prediction error/uncertainty of the model.

In practice, the second option is usually preferred. In that scenario, the prediction output will not just be a single value. Instead, we will get a full probability distribution of it. This is…