Kolmogorov–Arnold Networks

KANs (Kolmogorov–Arnold Networks) were proposed by Liu et al. (2024) as a generalization of the Kolmogorov–Arnold representation theorem (KART), aiming to outperform MLPs in small-scale AI and scientific tasks. Before KANs, numerous studies explored KART's connections to neural networks or used it as a basis for designing new network architectures. In the 1980s and 1990s, early research applied KART to neural network design. Kůrková et al. (1992), Hecht-Nielsen (1987), and Nees (1994) established theoretical foundations for multilayer networks based on KART. Igelnik et al. (2003) introduced the Kolmogorov Spline Network using cubic splines to model complex functions. Sprecher (1996, 1997) introduced numerical methods for building network layers, while Nakamura et al. (1993) created activation functions with guaranteed approximation accuracy. These works linked KART's theoretical potential with practical neural network implementation. KART has also been used in other computational and theoretical fields. Coppejans (2004) developed nonparametric regression estimators using B-splines, Bryant (2008) applied it to high-dimensional image tasks, Liu (2015) investigated theoretical applications in optimal transport and image encryption, and more recently, Polar and Poluektov (2021) used Urysohn operators for efficient KART construction, while Fakhoury et al. (2022) introduced ExSpliNet, integrating KART with probabilistic trees and multivariate B-splines for improved function approximation. == Architecture ==

KANs are based on the Kolmogorov–Arnold representation theorem, which was linked to the 13th Hilbert problem. Given x = (x_1, x_2, \dots, x_n) consisting of n variables, a multivariate continuous function f(x) can be represented as: : f(x) = f(x_1, \dots, x_n) = \sum_{q=1}^{2n+1} \Phi_q \left( \sum_{p=1}^{n} \varphi_{q,p}(x_p) \right)   (1) This formulation contains two nested summations: an outer and an inner sum. The outer sum \sum_{q=1}^{2n+1} aggregates 2n+1 terms, each involving a function \Phi_q : \mathbb{R} \to \mathbb{R}. The inner sum \sum_{p=1}^n computes n terms for each q, where each term \varphi_{q,p} : [0,1] \to \mathbb{R} is a continuous function of the single variable x_p. The inner continuous functions \varphi_{q,p} are universal, independent of f, while the outer functions \Phi_q depend on the specific function f being represented. The representation (1) holds for all multivariate functions f as proved in . If f is continuous, then the outer functions \Phi_q are continuous; if f is discontinuous, then the corresponding \Phi_q are generally discontinuous, while the inner functions \varphi_{q,p} remain the same universal functions. Liu et al. proposed the name KAN. A general KAN network consisting of L layers takes x to generate the output as: :\mathrm{KAN}(x) = (\Phi^{L-1} \circ \Phi^{L-2} \circ \cdots \circ \Phi^{1} \circ \Phi^{0})x   (3) Here, \Phi^{l} is the function matrix of the l-th KAN layer or a set of pre-activations. Let i denote the neuron of the l-th layer and j the neuron of the (l+1)-th layer. The activation function \varphi^{l}_{j,i} connects (l, i) to (l+1, j): :\varphi^{l}_{j,i}, \quad l = 0,\dots,L-1, \; i = 1,\dots,n_l, \; j = 1,\dots,n_{l+1}   (4) where nl is the number of nodes of the l-th layer. Thus, the function matrix \Phi^{l} can be represented as an n_{l+1} \times n_l matrix of activations: : x^{l+1} = \begin{pmatrix} \varphi^{l}_{1,1}(\cdot) & \varphi^{l}_{1,2}(\cdot) & \cdots & \varphi^{l}_{1,n_l}(\cdot) \\ \varphi^{l}_{2,1}(\cdot) & \varphi^{l}_{2,2}(\cdot) & \cdots & \varphi^{l}_{2,n_l}(\cdot) \\ \vdots & \vdots & \ddots & \vdots \\ \varphi^{l}_{n_{l+1},1}(\cdot) & \varphi^{l}_{n_{l+1},2}(\cdot) & \cdots & \varphi^{l}_{n_{l+1},n_l}(\cdot) \end{pmatrix}x^{l} == Implementations ==

To make the KAN layers optimizable, the inner function is formed by the combination of spline and basic functions as the formula: == Functions used ==

The choice of functional basis strongly influences the performance of KANs. Common function families include: • B-splines: Provide locality, smoothness, and interpretability; they are the most widely used in current implementations. • Chebyshev polynomials: Offer efficient approximation with minimized error in the maximum norm, making them useful for stable function representation. • Rational function: Useful for approximating functions with singularities or sharp variations, as they can model asymptotic behavior better than polynomials. • Fourier series: Capture periodic patterns effectively and are particularly useful in domains such as physics-informed machine learning. • Wavelet functions (DoG, Mexican hat, Morlet, and Shannon): Used for feature extraction as they can capture both high-frequency and low-frequency data components. • Piecewise linear functions: Provide efficient approximation for multivariate functions in KANs. == Usage ==

In some modern neural architectures like convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformers, KANs are typically used as drop-in substitutes for MLP layers. Despite KANs' general-purpose design, researchers have created and used them for a number of tasks: • Scientific machine learning (SciML): Function fitting, and physical/mathematical laws. • Graph neural networks: Extensions such as Kolmogorov–Arnold Graph Neural Networks (KA-GNNs) integrate KAN modules into message-passing architectures, showing improvements in molecular property prediction tasks. == Drawbacks ==

KANs can be computationally intensive and require a large number of parameters due to their use of polynomial functions to capture data. == See also ==