Image by Author
A loss function in machine learning is a mathematical formula that calculates the difference between the predicted output and the actual output of the model. The loss function is then used to slightly change the model weights and then check whether it has improved the model’s performance. The goal of machine learning algorithms is to minimize the loss function in order to make accurate predictions.
In this blog, we will learn about the 5 most commonly used loss functions for classification and regression machine learning algorithms.Â
Â
1. Binary Cross-Entropy Loss
Binary cross-entropy loss, or Log loss, is a commonly used loss function for binary classification. It calculates the difference between the predicted probabilities and the actual labels. Binary cross-entropy loss is widely used for spam detection, sentiment analysis, or cancer detection, where the goal is to distinguish between two classes.
The Binary Cross-Entropy loss function is defined as:
where y is the actual label (0 or 1), and ŷ is the predicted probability.
In this formula, the loss function penalizes the model based on how far the predicted probability ŷ is from the actual target value y.
2. Hinge Loss
Hinge loss is another loss function generally used for classification problems. It is often associated with Support Vector Machines (SVMs). Hinge loss calculates the difference between the predicted output and the actual label with a margin.
The Hinge loss function is defined as:
where y is the true label (+1 or -1), and ŷ is the predicted output.
The idea behind the Hinge loss is to penalize the model for misclassifications and being overly confident in its predictions.
3. Mean Square Error
Mean Square Error (MSE) is the most common loss function used for regression problems. It calculates the average squared difference between predicted and actual values.
The MSE loss function is defined as:
where:
- n is the number of samples.
- y1 is the true value of the i-th sample.
- ŷi is the predicted value of the i-th sample.
- Σ is the sum over all samples.
The Mean square error is a measure of the quality of an algorithm. It is always non-negative, and values closer to zero are better. It is sensitive to outliers, meaning that a single very wrong prediction can significantly increase the loss.
4. Mean Absolute Error
Mean Absolute Error (MAE) is another commonly used loss function for regression problems. It calculates the average absolute difference between predicted and actual values.
The MAE loss function is defined as:
where:
- n is the number of samples.
- yi is the true value of the i-th sample.
- ŷi is the predicted value of the i-th sample.
- Σ is the sum over all samples.
Similar to MSE, it is always non-negative, and values closer to zero are better. However, unlike the MSE, the MAE is less sensitive to outliers, meaning that a single very wrong prediction won’t significantly increase the loss.
5. Huber Loss
Huber loss, also known as smooth mean absolute error, is a combination of Mean Square Error and Mean Absolute Error, making it a useful loss function for regression tasks, especially when dealing with noisy data.
The Huber loss function is defined as:
where:
- y is the actual value.
- ŷ is the predicted value.
- δ is a hyperparameter that controls the sensitivity to outliers.
If the loss values are less than δ, use the MSE; if the loss values are greater than δ, use the MAE. It combines the best of both worlds from the two high-performance loss functions.Â
MSE is excellent for detecting outliers, whereas MAE is great for ignoring them; Huber loss offers a balance between the two.
Conclusion
Just like how car headlights illuminate the road ahead, helping us navigate through the darkness and reach our destination safely, a loss function provides guidance to a machine learning algorithm, helping it navigate through the complex landscape of possible solutions and reach its optimal performance. This guidance helps in making adjustments to the model parameters to minimize error and improve accuracy, thereby steering the algorithm towards its optimal performance.
In this blog, we have learned about 2 classification (Binary Cross-Entropy, Hinge) and 3 regression (Mean Square Error, Mean Absolute Error, Huber) loss functions. They are all popular functions for calculating the difference between predicted and actual values.
About Abid Ali Awan
Abid Ali Awan is the Assistant Editor of KDnuggets. Abid is a certified Data Scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master’s degree in technology management and a bachelor’s degree in telecommunication Engineering.