Get it on Google Play
New! Download Unionpedia on your Android™ device!
Faster access than browser!

Gradient descent

Index Gradient descent

Gradient descent is a first-order iterative optimization algorithm for finding the minimum of a function. [1]

63 relations: Accuracy and precision, Algorithm, Artificial neural network, Backpropagation, Big O notation, Bowl, Broyden–Fletcher–Goldfarb–Shanno algorithm, Concentric objects, Conjugate gradient method, Constraint (mathematics), Contour line, Convex function, Convex optimization, Coursera, Curvature, Delta rule, Derivative, Differentiable function, Differential calculus, Eigenvalues and eigenvectors, Euler method, Fréchet derivative, Function of several real variables, Function space, Gauss–Newton algorithm, Geoffrey Hinton, Gradient, Hessian matrix, Hill climbing, Iterative method, Jacobian matrix and determinant, Limited-memory BFGS, Line search, Linear least squares (mathematics), Lipschitz continuity, Mathematical analysis, Mathematical optimization, MATLAB, Maxima and minima, Method of steepest descent, Nelder–Mead method, Neural Networks (journal), Newton's method in optimization, Norm (mathematics), Ordinary differential equation, Orthogonality, Preconditioner, Projection (linear algebra), Proximal gradient method, Python (programming language), ..., Regina S. Burachik, Rosenbrock function, Rprop, Slope, Stochastic gradient descent, Subgradient method, Undefined (mathematics), Variational inequality, Vector field, Willamette University, Wolfe conditions, YouTube, Yurii Nesterov. Expand index (13 more) »

Accuracy and precision

Precision is a description of random errors, a measure of statistical variability.

New!!: Gradient descent and Accuracy and precision · See more »


In mathematics and computer science, an algorithm is an unambiguous specification of how to solve a class of problems.

New!!: Gradient descent and Algorithm · See more »

Artificial neural network

Artificial neural networks (ANNs) or connectionist systems are computing systems vaguely inspired by the biological neural networks that constitute animal brains.

New!!: Gradient descent and Artificial neural network · See more »


Backpropagation is a method used in artificial neural networks to calculate a gradient that is needed in the calculation of the weights to be used in the network.

New!!: Gradient descent and Backpropagation · See more »

Big O notation

Big O notation is a mathematical notation that describes the limiting behaviour of a function when the argument tends towards a particular value or infinity.

New!!: Gradient descent and Big O notation · See more »


A bowl is a round, open-top container used in many cultures to serve hot and cold food.

New!!: Gradient descent and Bowl · See more »

Broyden–Fletcher–Goldfarb–Shanno algorithm

In numerical optimization, the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm is an iterative method for solving unconstrained nonlinear optimization problems.

New!!: Gradient descent and Broyden–Fletcher–Goldfarb–Shanno algorithm · See more »

Concentric objects

In geometry, two or more objects are said to be concentric, coaxal, or coaxial when they share the same center or axis.

New!!: Gradient descent and Concentric objects · See more »

Conjugate gradient method

In mathematics, the conjugate gradient method is an algorithm for the numerical solution of particular systems of linear equations, namely those whose matrix is symmetric and positive-definite.

New!!: Gradient descent and Conjugate gradient method · See more »

Constraint (mathematics)

In mathematics, a constraint is a condition of an optimization problem that the solution must satisfy.

New!!: Gradient descent and Constraint (mathematics) · See more »

Contour line

A contour line (also isocline, isopleth, isarithm, or equipotential curve) of a function of two variables is a curve along which the function has a constant value, so that the curve joins points of equal value.

New!!: Gradient descent and Contour line · See more »

Convex function

In mathematics, a real-valued function defined on an ''n''-dimensional interval is called convex (or convex downward or concave upward) if the line segment between any two points on the graph of the function lies above or on the graph, in a Euclidean space (or more generally a vector space) of at least two dimensions.

New!!: Gradient descent and Convex function · See more »

Convex optimization

Convex optimization is a subfield of optimization that studies the problem of minimizing convex functions over convex sets.

New!!: Gradient descent and Convex optimization · See more »


Coursera is an online learning platform founded by Stanford professors Andrew Ng and Daphne Koller that offers courses, specializations, and degrees.

New!!: Gradient descent and Coursera · See more »


In mathematics, curvature is any of a number of loosely related concepts in different areas of geometry.

New!!: Gradient descent and Curvature · See more »

Delta rule

In machine learning, the Delta rule is a gradient descent learning rule for updating the weights of the inputs to artificial neurons in a single-layer neural network.

New!!: Gradient descent and Delta rule · See more »


The derivative of a function of a real variable measures the sensitivity to change of the function value (output value) with respect to a change in its argument (input value).

New!!: Gradient descent and Derivative · See more »

Differentiable function

In calculus (a branch of mathematics), a differentiable function of one real variable is a function whose derivative exists at each point in its domain.

New!!: Gradient descent and Differentiable function · See more »

Differential calculus

In mathematics, differential calculus is a subfield of calculus concerned with the study of the rates at which quantities change.

New!!: Gradient descent and Differential calculus · See more »

Eigenvalues and eigenvectors

In linear algebra, an eigenvector or characteristic vector of a linear transformation is a non-zero vector that changes by only a scalar factor when that linear transformation is applied to it.

New!!: Gradient descent and Eigenvalues and eigenvectors · See more »

Euler method

In mathematics and computational science, the Euler method (also called forward Euler method) is a first-order numerical procedure for solving ordinary differential equations (ODEs) with a given initial value.

New!!: Gradient descent and Euler method · See more »

Fréchet derivative

In mathematics, the Fréchet derivative is a derivative defined on Banach spaces.

New!!: Gradient descent and Fréchet derivative · See more »

Function of several real variables

In mathematical analysis, and applications in geometry, applied mathematics, engineering, natural sciences, and economics, a function of several real variables or real multivariate function is a function with more than one argument, with all arguments being real variables.

New!!: Gradient descent and Function of several real variables · See more »

Function space

In mathematics, a function space is a set of functions between two fixed sets.

New!!: Gradient descent and Function space · See more »

Gauss–Newton algorithm

The Gauss–Newton algorithm is used to solve non-linear least squares problems.

New!!: Gradient descent and Gauss–Newton algorithm · See more »

Geoffrey Hinton

Geoffrey Everest Hinton One or more of the preceding sentences incorporates text from the royalsociety.org website where: (born 6 December 1947) is a British cognitive psychologist and computer scientist, most noted for his work on artificial neural networks.

New!!: Gradient descent and Geoffrey Hinton · See more »


In mathematics, the gradient is a multi-variable generalization of the derivative.

New!!: Gradient descent and Gradient · See more »

Hessian matrix

In mathematics, the Hessian matrix or Hessian is a square matrix of second-order partial derivatives of a scalar-valued function, or scalar field.

New!!: Gradient descent and Hessian matrix · See more »

Hill climbing

In numerical analysis, hill climbing is a mathematical optimization technique which belongs to the family of local search.

New!!: Gradient descent and Hill climbing · See more »

Iterative method

In computational mathematics, an iterative method is a mathematical procedure that uses an initial guess to generate a sequence of improving approximate solutions for a class of problems, in which the n-th approximation is derived from the previous ones.

New!!: Gradient descent and Iterative method · See more »

Jacobian matrix and determinant

In vector calculus, the Jacobian matrix is the matrix of all first-order partial derivatives of a vector-valued function.

New!!: Gradient descent and Jacobian matrix and determinant · See more »

Limited-memory BFGS

Limited-memory BFGS (L-BFGS or LM-BFGS) is an optimization algorithm in the family of quasi-Newton methods that approximates the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm using a limited amount of computer memory.

New!!: Gradient descent and Limited-memory BFGS · See more »

Line search

In optimization, the line search strategy is one of two basic iterative approaches to find a local minimum \mathbf^* of an objective function f:\mathbb R^n\to\mathbb R. The other approach is trust region.

New!!: Gradient descent and Line search · See more »

Linear least squares (mathematics)

In statistics and mathematics, linear least squares is an approach to fitting a mathematical or statistical model to data in cases where the idealized value provided by the model for any data point is expressed linearly in terms of the unknown parameters of the model.

New!!: Gradient descent and Linear least squares (mathematics) · See more »

Lipschitz continuity

In mathematical analysis, Lipschitz continuity, named after Rudolf Lipschitz, is a strong form of uniform continuity for functions.

New!!: Gradient descent and Lipschitz continuity · See more »

Mathematical analysis

Mathematical analysis is the branch of mathematics dealing with limits and related theories, such as differentiation, integration, measure, infinite series, and analytic functions.

New!!: Gradient descent and Mathematical analysis · See more »

Mathematical optimization

In mathematics, computer science and operations research, mathematical optimization or mathematical programming, alternatively spelled optimisation, is the selection of a best element (with regard to some criterion) from some set of available alternatives.

New!!: Gradient descent and Mathematical optimization · See more »


MATLAB (matrix laboratory) is a multi-paradigm numerical computing environment and proprietary programming language developed by MathWorks.

New!!: Gradient descent and MATLAB · See more »

Maxima and minima

In mathematical analysis, the maxima and minima (the respective plurals of maximum and minimum) of a function, known collectively as extrema (the plural of extremum), are the largest and smallest value of the function, either within a given range (the local or relative extrema) or on the entire domain of a function (the global or absolute extrema).

New!!: Gradient descent and Maxima and minima · See more »

Method of steepest descent

In mathematics, the method of steepest descent or stationary-phase method or saddle-point method is an extension of Laplace's method for approximating an integral, where one deforms a contour integral in the complex plane to pass near a stationary point (saddle point), in roughly the direction of steepest descent or stationary phase.

New!!: Gradient descent and Method of steepest descent · See more »

Nelder–Mead method

The Nelder–Mead method or downhill simplex method or amoeba method is a commonly applied numerical method used to find the minimum or maximum of an objective function in a multidimensional space.

New!!: Gradient descent and Nelder–Mead method · See more »

Neural Networks (journal)

Neural Networks is a monthly peer-reviewed scientific journal and an official journal of the International Neural Network Society, European Neural Network Society, and Japanese Neural Network Society.

New!!: Gradient descent and Neural Networks (journal) · See more »

Newton's method in optimization

In calculus, Newton's method is an iterative method for finding the roots of a differentiable function (i.e. solutions to the equation). In optimization, Newton's method is applied to the derivative of a twice-differentiable function to find the roots of the derivative (solutions to), also known as the stationary points of.

New!!: Gradient descent and Newton's method in optimization · See more »

Norm (mathematics)

In linear algebra, functional analysis, and related areas of mathematics, a norm is a function that assigns a strictly positive length or size to each vector in a vector space—save for the zero vector, which is assigned a length of zero.

New!!: Gradient descent and Norm (mathematics) · See more »

Ordinary differential equation

In mathematics, an ordinary differential equation (ODE) is a differential equation containing one or more functions of one independent variable and its derivatives.

New!!: Gradient descent and Ordinary differential equation · See more »


In mathematics, orthogonality is the generalization of the notion of perpendicularity to the linear algebra of bilinear forms.

New!!: Gradient descent and Orthogonality · See more »


In mathematics, preconditioning is the application of a transformation, called the preconditioner, that conditions a given problem into a form that is more suitable for numerical solving methods.

New!!: Gradient descent and Preconditioner · See more »

Projection (linear algebra)

In linear algebra and functional analysis, a projection is a linear transformation P from a vector space to itself such that.

New!!: Gradient descent and Projection (linear algebra) · See more »

Proximal gradient method

Proximal gradient methods are a generalized form of projection used to solve non-differentiable convex optimization problems.

New!!: Gradient descent and Proximal gradient method · See more »

Python (programming language)

Python is an interpreted high-level programming language for general-purpose programming.

New!!: Gradient descent and Python (programming language) · See more »

Regina S. Burachik

Regina Sandra Burachik is an Argentine mathematician who works on optimization and analysis (particularly: convex analysis, functional analysis and non-smooth analysis).

New!!: Gradient descent and Regina S. Burachik · See more »

Rosenbrock function

In mathematical optimization, the Rosenbrock function is a non-convex function, introduced by Howard H. Rosenbrock in 1960, which is used as a performance test problem for optimization algorithms.

New!!: Gradient descent and Rosenbrock function · See more »


Rprop, short for resilient backpropagation, is a learning heuristic for supervised learning in feedforward artificial neural networks.

New!!: Gradient descent and Rprop · See more »


In mathematics, the slope or gradient of a line is a number that describes both the direction and the steepness of the line.

New!!: Gradient descent and Slope · See more »

Stochastic gradient descent

Stochastic gradient descent (often shortened to SGD), also known as incremental gradient descent, is an iterative method for optimizing a differentiable objective function, a stochastic approximation of gradient descent optimization.

New!!: Gradient descent and Stochastic gradient descent · See more »

Subgradient method

Subgradient methods are iterative methods for solving convex minimization problems.

New!!: Gradient descent and Subgradient method · See more »

Undefined (mathematics)

In mathematics, undefined has several different meanings, depending on the context.

New!!: Gradient descent and Undefined (mathematics) · See more »

Variational inequality

In mathematics, a variational inequality is an inequality involving a functional, which has to be solved for all possible values of a given variable, belonging usually to a convex set.

New!!: Gradient descent and Variational inequality · See more »

Vector field

In vector calculus and physics, a vector field is an assignment of a vector to each point in a subset of space.

New!!: Gradient descent and Vector field · See more »

Willamette University

Willamette University is a private liberal arts college located in Salem, Oregon, United States.

New!!: Gradient descent and Willamette University · See more »

Wolfe conditions

In the unconstrained minimization problem, the Wolfe conditions are a set of inequalities for performing inexact line search, especially in quasi-Newton methods, first published by Philip Wolfe in 1969.

New!!: Gradient descent and Wolfe conditions · See more »


YouTube is an American video-sharing website headquartered in San Bruno, California.

New!!: Gradient descent and YouTube · See more »

Yurii Nesterov

Yurii Nesterov is a Russian mathematician, an internationally recognized expert in convex optimization, especially in the development of efficient algorithms and numerical optimization analysis.

New!!: Gradient descent and Yurii Nesterov · See more »

Redirects here:

Gradient Descent, Gradient ascent, Gradient descent method, Gradient descent optimization, Steepest ascent, Steepest descent.


[1] https://en.wikipedia.org/wiki/Gradient_descent

Hey! We are on Facebook now! »