[{"data":1,"prerenderedAt":557},["ShallowReactive",2],{"$uzWF-xL96Z":3},{"id":4,"title":5,"body":6,"description":58,"extension":550,"meta":551,"navigation":552,"path":553,"seo":554,"stem":555,"__hash__":556},"content/modules/optimization/gradient-descent/content.md","Gradient Descent: The Foundation of Machine Learning Optimization",{"type":7,"value":8,"toc":521},"minimark",[9,13,18,22,25,29,40,45,48,59,62,85,88,92,95,115,118,122,125,129,132,137,147,151,160,163,169,172,183,189,193,196,202,205,219,224,228,231,237,240,254,260,264,273,276,280,283,289,293,296,301,305,308,328,334,338,341,346,350,354,357,377,381,384,404,408,435,439,442,468,472,492,496],[10,11,5],"h1",{"id":12},"gradient-descent-the-foundation-of-machine-learning-optimization",[14,15,17],"h2",{"id":16},"introduction","Introduction",[19,20,21],"p",{},"Gradient descent is the workhorse optimization algorithm behind most machine learning models. Whether you're training a simple linear regression or a complex deep neural network, gradient descent (or one of its variants) is likely doing the heavy lifting to find the best parameters.",[19,23,24],{},"At its core, gradient descent is beautifully simple: it iteratively adjusts parameters in the direction that most reduces the loss function. Think of it like hiking down a mountain in fog—you can't see the bottom, but you can feel which direction is steepest downward, so you take small steps in that direction.",[14,26,28],{"id":27},"the-gradient-descent-algorithm","The Gradient Descent Algorithm",[19,30,31,36],{},[32,33],"img",{"alt":34,"src":35},"Gradient Descent Optimization","https://upload.wikimedia.org/wikipedia/commons/thumb/a/a3/Gradient_descent.gif/350px-Gradient_descent.gif",[37,38,39],"em",{},"Gradient descent iteratively moves toward the minimum of the loss function",[41,42,44],"h3",{"id":43},"basic-concept","Basic Concept",[19,46,47],{},"Given a loss function L(θ) that measures how well our model performs, gradient descent updates the parameters θ using this rule:",[49,50,55],"pre",{"className":51,"code":53,"language":54},[52],"language-text","θ = θ - α∇L(θ)\n","text",[56,57,53],"code",{"__ignoreMap":58},"",[19,60,61],{},"Where:",[63,64,65,73,79],"ul",{},[66,67,68,72],"li",{},[69,70,71],"strong",{},"θ"," represents the model parameters (weights, biases, etc.)",[66,74,75,78],{},[69,76,77],{},"α"," is the learning rate (step size)",[66,80,81,84],{},[69,82,83],{},"∇L(θ)"," is the gradient of the loss function (direction of steepest ascent)",[19,86,87],{},"The negative sign means we move in the opposite direction of the gradient—downhill toward lower loss.",[41,89,91],{"id":90},"the-learning-rate","The Learning Rate",[19,93,94],{},"The learning rate α is crucial:",[63,96,97,103,109],{},[66,98,99,102],{},[69,100,101],{},"Too small",": Convergence is slow, requiring many iterations",[66,104,105,108],{},[69,106,107],{},"Too large",": May overshoot the minimum or even diverge",[66,110,111,114],{},[69,112,113],{},"Just right",": Efficient convergence to a good solution",[19,116,117],{},"Finding the right learning rate often requires experimentation.",[14,119,121],{"id":120},"optimizer-variants","Optimizer Variants",[19,123,124],{},"While basic gradient descent works, researchers have developed sophisticated variants that converge faster and more reliably.",[41,126,128],{"id":127},"sgd-stochastic-gradient-descent","SGD (Stochastic Gradient Descent)",[19,130,131],{},"The simplest form—just follow the gradient:",[49,133,135],{"className":134,"code":53,"language":54},[52],[56,136,53],{"__ignoreMap":58},[19,138,139,142,143,146],{},[69,140,141],{},"Pros",": Simple, easy to understand\n",[69,144,145],{},"Cons",": Can be slow, sensitive to learning rate, oscillates in ravines",[41,148,150],{"id":149},"sgd-with-momentum","SGD with Momentum",[19,152,153,157],{},[32,154],{"alt":155,"src":156},"Momentum Visualization","https://upload.wikimedia.org/wikipedia/commons/thumb/f/f3/Stogra.png/500px-Stogra.png",[37,158,159],{},"Momentum helps accelerate convergence and reduce oscillations",[19,161,162],{},"Adds \"momentum\" to accelerate convergence and dampen oscillations:",[49,164,167],{"className":165,"code":166,"language":54},[52],"v = βv + ∇L(θ)\nθ = θ - αv\n",[56,168,166],{"__ignoreMap":58},[19,170,171],{},"The velocity v accumulates gradients over time, helping the optimizer:",[63,173,174,177,180],{},[66,175,176],{},"Build speed in consistent directions",[66,178,179],{},"Dampen oscillations in inconsistent directions",[66,181,182],{},"Escape shallow local minima",[19,184,185,188],{},[69,186,187],{},"Typical β value",": 0.9 (90% of previous velocity retained)",[41,190,192],{"id":191},"rmsprop-root-mean-square-propagation","RMSProp (Root Mean Square Propagation)",[19,194,195],{},"Adapts the learning rate for each parameter based on recent gradient magnitudes:",[49,197,200],{"className":198,"code":199,"language":54},[52],"E[g²] = βE[g²] + (1-β)g²\nθ = θ - α·g / √(E[g²] + ε)\n",[56,201,199],{"__ignoreMap":58},[19,203,204],{},"This helps with:",[63,206,207,213],{},[66,208,209,212],{},[69,210,211],{},"Ill-conditioned problems",": Different parameters need different learning rates",[66,214,215,218],{},[69,216,217],{},"Ravines",": Steep in some directions, shallow in others",[19,220,221,223],{},[69,222,187],{},": 0.9",[41,225,227],{"id":226},"adam-adaptive-moment-estimation","Adam (Adaptive Moment Estimation)",[19,229,230],{},"Combines the best of momentum and RMSProp:",[49,232,235],{"className":233,"code":234,"language":54},[52],"m = β₁m + (1-β₁)g          # First moment (momentum)\nv = β₂v + (1-β₂)g²         # Second moment (RMSProp)\nm̂ = m / (1-β₁ᵗ)            # Bias correction\nv̂ = v / (1-β₂ᵗ)            # Bias correction\nθ = θ - α·m̂ / (√v̂ + ε)\n",[56,236,234],{"__ignoreMap":58},[19,238,239],{},"Adam is often the default choice because it:",[63,241,242,245,248,251],{},[66,243,244],{},"Adapts learning rates per parameter",[66,246,247],{},"Includes momentum for acceleration",[66,249,250],{},"Corrects for initialization bias",[66,252,253],{},"Works well across many problems",[19,255,256,259],{},[69,257,258],{},"Typical values",": β₁=0.9, β₂=0.999, ε=1e-8",[14,261,263],{"id":262},"loss-landscapes","Loss Landscapes",[19,265,266,270],{},[32,267],{"alt":268,"src":269},"Loss Landscape Visualization","https://upload.wikimedia.org/wikipedia/commons/thumb/1/1e/Extrema_example_original.svg/500px-Extrema_example_original.svg.png",[37,271,272],{},"Different types of loss landscapes with local minima and saddle points",[19,274,275],{},"Understanding loss landscapes helps explain optimizer behavior.",[41,277,279],{"id":278},"convex-functions","Convex Functions",[19,281,282],{},"Simple bowl-shaped functions have one global minimum. All optimizers easily find it, though at different speeds.",[19,284,285,288],{},[69,286,287],{},"Example",": Quadratic bowl (x² + y²)",[41,290,292],{"id":291},"ill-conditioned-functions","Ill-Conditioned Functions",[19,294,295],{},"Elongated valleys where gradients are much steeper in some directions. Basic SGD oscillates; momentum and adaptive methods handle these better.",[19,297,298,300],{},[69,299,287],{},": Elongated valley (x² + 10y²)",[41,302,304],{"id":303},"non-convex-functions","Non-Convex Functions",[19,306,307],{},"Real-world loss landscapes with:",[63,309,310,316,322],{},[66,311,312,315],{},[69,313,314],{},"Local minima",": Suboptimal solutions that trap basic optimizers",[66,317,318,321],{},[69,319,320],{},"Saddle points",": Flat regions that slow convergence",[66,323,324,327],{},[69,325,326],{},"Plateaus",": Nearly flat regions with tiny gradients",[19,329,330,333],{},[69,331,332],{},"Examples",": Rosenbrock function, Himmelblau function",[41,335,337],{"id":336},"multi-modal-functions","Multi-Modal Functions",[19,339,340],{},"Multiple local minima of varying quality. The starting point matters!",[19,342,343,345],{},[69,344,287],{},": Rastrigin function (many local minima)",[14,347,349],{"id":348},"practical-considerations","Practical Considerations",[41,351,353],{"id":352},"convergence-criteria","Convergence Criteria",[19,355,356],{},"When to stop optimizing?",[63,358,359,365,371],{},[66,360,361,364],{},[69,362,363],{},"Gradient norm",": Stop when ||∇L(θ)|| \u003C ε (gradient is tiny)",[66,366,367,370],{},[69,368,369],{},"Loss change",": Stop when loss improvement is negligible",[66,372,373,376],{},[69,374,375],{},"Maximum iterations",": Prevent infinite loops",[41,378,380],{"id":379},"hyperparameter-tuning","Hyperparameter Tuning",[19,382,383],{},"Key hyperparameters to tune:",[63,385,386,392,398],{},[66,387,388,391],{},[69,389,390],{},"Learning rate (α)",": Most important; try 0.001, 0.01, 0.1",[66,393,394,397],{},[69,395,396],{},"Momentum (β)",": Usually 0.9 or 0.99",[66,399,400,403],{},[69,401,402],{},"Batch size",": Affects gradient noise (not shown in this visualization)",[41,405,407],{"id":406},"common-pitfalls","Common Pitfalls",[409,410,411,417,423,429],"ol",{},[66,412,413,416],{},[69,414,415],{},"Exploding gradients",": Learning rate too high, parameters diverge",[66,418,419,422],{},[69,420,421],{},"Vanishing gradients",": Gradients become too small to make progress",[66,424,425,428],{},[69,426,427],{},"Poor initialization",": Starting in a bad region of the loss landscape",[66,430,431,434],{},[69,432,433],{},"Wrong optimizer",": Some problems need specific optimizers",[14,436,438],{"id":437},"interactive-exploration","Interactive Exploration",[19,440,441],{},"Use the interactive visualization to:",[409,443,444,450,456,462],{},[66,445,446,449],{},[69,447,448],{},"Compare optimizers",": See how SGD, Momentum, RMSProp, and Adam navigate the same landscape",[66,451,452,455],{},[69,453,454],{},"Adjust learning rate",": Observe the impact on convergence speed and stability",[66,457,458,461],{},[69,459,460],{},"Try different landscapes",": See how optimizer performance varies by problem type",[66,463,464,467],{},[69,465,466],{},"Step through optimization",": Watch the path unfold iteration by iteration",[14,469,471],{"id":470},"key-takeaways","Key Takeaways",[63,473,474,477,480,483,486,489],{},[66,475,476],{},"Gradient descent is the foundation of ML optimization",[66,478,479],{},"The learning rate is the most critical hyperparameter",[66,481,482],{},"Momentum helps accelerate convergence and escape local minima",[66,484,485],{},"Adaptive methods (RMSProp, Adam) adjust learning rates per parameter",[66,487,488],{},"Loss landscape shape dramatically affects optimizer performance",[66,490,491],{},"Adam is often a good default choice for most problems",[14,493,495],{"id":494},"further-reading","Further Reading",[63,497,498,507,514],{},[66,499,500],{},[501,502,506],"a",{"href":503,"rel":504},"https://ruder.io/optimizing-gradient-descent/",[505],"nofollow","An overview of gradient descent optimization algorithms",[66,508,509],{},[501,510,513],{"href":511,"rel":512},"https://arxiv.org/abs/1412.6980",[505],"Adam: A Method for Stochastic Optimization",[66,515,516],{},[501,517,520],{"href":518,"rel":519},"http://proceedings.mlr.press/v28/sutskever13.html",[505],"On the importance of initialization and momentum in deep learning",{"title":58,"searchDepth":522,"depth":523,"links":524},2,3,[525,526,530,536,542,547,548,549],{"id":16,"depth":522,"text":17},{"id":27,"depth":522,"text":28,"children":527},[528,529],{"id":43,"depth":523,"text":44},{"id":90,"depth":523,"text":91},{"id":120,"depth":522,"text":121,"children":531},[532,533,534,535],{"id":127,"depth":523,"text":128},{"id":149,"depth":523,"text":150},{"id":191,"depth":523,"text":192},{"id":226,"depth":523,"text":227},{"id":262,"depth":522,"text":263,"children":537},[538,539,540,541],{"id":278,"depth":523,"text":279},{"id":291,"depth":523,"text":292},{"id":303,"depth":523,"text":304},{"id":336,"depth":523,"text":337},{"id":348,"depth":522,"text":349,"children":543},[544,545,546],{"id":352,"depth":523,"text":353},{"id":379,"depth":523,"text":380},{"id":406,"depth":523,"text":407},{"id":437,"depth":522,"text":438},{"id":470,"depth":522,"text":471},{"id":494,"depth":522,"text":495},"md",{},true,"/modules/optimization/gradient-descent/content",{"title":5,"description":58},"modules/optimization/gradient-descent/content","MwkdwEqz8RfXMbe0So4nEO1htPrr5A0UoTUfi56iAS0",1792223550791]