In the case of
backpropagation based
artificial neural networks or
perceptrons, the type of decision boundary that the network can learn is determined by the number of hidden layers the network has. If it has no hidden layers, then it can only learn linear problems. If it has one hidden layer, then it can learn any
continuous function on
compact subsets of
Rn as shown by the
universal approximation theorem, thus it can have an arbitrary decision boundary. In particular,
support vector machines find a
hyperplane that separates the feature space into two classes with the
maximum margin. If the problem is not originally linearly separable, the
kernel trick can be used to turn it into a linearly separable one, by increasing the number of dimensions. Thus a general hypersurface in a small dimension space is turned into a hyperplane in a space with much larger dimensions. Neural networks try to learn the decision boundary which minimizes the empirical error, while support vector machines try to learn the decision boundary which maximizes the empirical margin between the decision boundary and data points. ==See also==