You can’t always get what you want
Sometimes, we don’t get what we want. And sometimes, there are laws of nature that positively forbid us from getting it, no matter how hard we try.
The Cramér–Rao bound is one such law that fixes an indestructible glass ceiling — a hard upper bound — on the predictive precision achieved by any regression model that anyone can think of.
In this article, we’ll study this law.
I’ll cover the following two topics:
- WHAT is the Cramér–Rao Bound? I’ll introduce both the variance and the precision forms of the bound. Piece-by-piece, I’ll unpack its definition, and I’ll explain how it all comes together.
- HOW is the Cramér–Rao Bound applied to regression models? Using a real life dataset and a linear regression model, we’ll study the applicability of the Cramér–Rao Bound to regression modeling.
The Cramér–Rao Bound (CRB) can be stated in two ways: as a lower bound on variance, and as an upper bound on precision.
When you read it for the first time, the definition of the Cramér–Rao Bound might feel like you are drinking from a hose. But don’t worry…