Loss Functions

Review: Loss functions evaluate how well the model performs (how much the predicted result deviates from actual result)

Regression Loss Functions

• Predicts continuous value (eg. floor size, # of rooms)

Mean Square Error / Quadratic Loss / L2 Loss

MSE

• Measures magnitude without considering direction

• Squaring penalizes further deviated values much more than less deviated values

Mean Absolute Error / L1 Loss

MAE

• More robust to outliers (no squaring)

• Needs linear programming to calculate gradients

Mean Bias Error

MBE

• Same as MAE but no absolute

  • This is kinda bad as it makes things less accurate

• Used for seeing positive or negative bias

Classification Loss Functions

• Predict output from finite set of categorical values (e.g. 0-9)

Hinge Loss / Multiclass SVM Loss

Hinge Loss

• Score of correct category should be greater than sum of scores of all incorrect categories by some safety margin (usually one)

• Usually for SVM

Cross Entropy Loss / Negative Log Likelihood

Cross Entropy Loss

• Most common

• Loss increases as predicted probability diverges from actual label

• Penalizes heavily predictions that are confident but WRONG

Cross Entropy Loss vs. KL Divergence

https://www.youtube.com/watch?v=pH9xkCK4ATc (great video!)