The concept of the cost function is at the base of machine learning algorithms. What is the intuition behind it?
Given a training set of input variables x(i) and output variables y(i), where y(i) is what we are trying to predict, a supervised learning problem is formally defined as h : X→Y, so that h(x) is a good predictor for the corresponding values of y. When the output variable y is continuous we call the learning problem a regression problem; if y can take only a small number of discrete values we call it a classification problem.
The cost function is what lets us figure out the most accurate line through our data points. The best possible line is the one for which the average squared vertical distance of the scattered points from the line is the least.
That quantity is the mean squared error, and it is what gradient descent is actually minimising. It is worth sitting with the picture of vertical distances before reaching for the formula, because every later loss function is a variation on the same question: how wrong is this prediction, and how should that wrongness be counted.
