Skip to content

Commit 6937503

Browse files
janbenderjanbender
committed
- added Newton_with_backtracking_line_search.html
1 parent 92315d9 commit 6937503

2 files changed

Lines changed: 379 additions & 0 deletions

File tree

‎examples/Newton_with_backtracking_line_search.html‎

Lines changed: 378 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,378 @@
1+
<!doctype html>
2+
<html class="no-js" lang="en">
3+
<head>
4+
<meta charset="utf-8">
5+
<meta name="viewport" content="width=device-width, initial-scale=1">
6+
<link rel="stylesheet" href="https://interactivecomputergraphics.github.io/physics-simulation/examples/style.css">
7+
<script type="text/x-mathjax-config">
8+
MathJax.Hub.Config({
9+
extensions: ["tex2jax.js"],
10+
jax: ["input/TeX", "output/HTML-CSS"],
11+
tex2jax: {
12+
inlineMath: [ ['$','$'], ["\\(","\\)"] ],
13+
displayMath: [ ['$$','$$'], ["\\[","\\]"] ],
14+
processEscapes: true
15+
},
16+
"HTML-CSS": { fonts: ["TeX"] }
17+
});
18+
</script>
19+
<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.4/MathJax.js"></script>
20+
<script src="https://cdn.plot.ly/plotly-2.5.1.min.js"></script>
21+
<title>Newton's Method with Backtracking Line Search</title>
22+
</head>
23+
<body>
24+
25+
<header class="page-header">
26+
<h1>Newton's Method with Backtracking Line Search</h1>
27+
</header>
28+
29+
<main>
30+
<!-- Simulation panel: canvas + controls -->
31+
<div class="card sim-panel">
32+
<div class="sim-canvas-wrap">
33+
<div id="plotOutput" style="width: 100%; height: auto;border:0px solid #000000;border-radius: 0px;background-color:#EEEEEE"></div>
34+
</div>
35+
</div>
36+
<div class="controls-panel" style="width: 100%;align:center;margin-left:auto;margin-right:auto">
37+
<h3>Controls</h3>
38+
<div class="controls-grid" style="width: 400px;align:left">
39+
<label>Newton steps</label>
40+
<p>
41+
<button onclick="plot.changeSteps(-1)" style="width:30px;height:30px;font-size:16px;cursor:pointer">&#8722;</button>
42+
<input type="text" id="textInput" value="1" readonly style="width:40px;text-align:center">
43+
<button onclick="plot.changeSteps(+1)" style="width:30px;height:30px;font-size:16px;cursor:pointer">+</button>
44+
</p>
45+
46+
<label for="armijo_c">Armijo parameter $c$</label>
47+
<input type="text" id="textInputC" value="0.50" readonly>
48+
49+
<label></label>
50+
<input onchange="document.getElementById('textInputC').value=parseFloat(this.value).toFixed(4);plot.reset()" id="armijo_c" value="0.50" type="range" min="0.0001" max="0.99" step="0.0001">
51+
52+
<label for="tau">Backtracking factor $\tau$</label>
53+
<input type="text" id="textInputTau" value="0.50" readonly>
54+
55+
<label></label>
56+
<input onchange="document.getElementById('textInputTau').value=parseFloat(this.value).toFixed(2);plot.reset()" id="tau" value="0.50" type="range" min="0.10" max="0.95" step="0.05">
57+
58+
<label for="fct">Function</label>
59+
<select onchange="plot.reset()" id="fct" size="1">
60+
<option selected="selected">Hyperbolic function</option>
61+
<option>Sinusoidal function</option>
62+
<option>Gaussian bump function</option>
63+
<option>Log-rational function</option>
64+
</select>
65+
</div>
66+
</div>
67+
68+
<!-- Theory section -->
69+
<div class="card theory">
70+
<h2>Newton's method with backtracking line search</h2>
71+
72+
<p>Newton's method for minimization uses the <em>Newton direction</em> $d_n = -f'(x_n)/f''(x_n)$ and updates:</p>
73+
$$x_{n+1} = x_n + \alpha_n \, d_n = x_n - \alpha_n \frac{f'(x_n)}{f''(x_n)}$$
74+
<p>With $\alpha_n = 1$ this recovers the standard Newton step. However, far from the minimum the full Newton step can overshoot. Backtracking line search selects $\alpha_n$ automatically to guarantee sufficient decrease.</p>
75+
76+
<h2>The Armijo condition</h2>
77+
78+
<p>For a general descent direction $d_n$, the <strong>Armijo condition</strong> requires:</p>
79+
$$f(x_n + \alpha\, d_n) \;\leq\; f(x_n) + c\,\alpha\, f'(x_n)\,d_n$$
80+
<p>Substituting the Newton direction $d_n = -f'(x_n)/f''(x_n)$ gives:</p>
81+
$$f\!\left(x_n - \alpha\frac{f'(x_n)}{f''(x_n)}\right) \;\leq\; f(x_n) - c\,\alpha\,\frac{[f'(x_n)]^2}{f''(x_n)}$$
82+
<p>Here $c \in (0,1)$ is a small constant (typically $c = 10^{-4}$ in practice; larger values are used here to make the effect visible).</p>
83+
84+
<h2>The backtracking algorithm</h2>
85+
86+
<p>Starting from $\alpha = 1$ (the full Newton step), the step is repeatedly shrunk by a factor $\tau \in (0,1)$ until the Armijo condition is satisfied:</p>
87+
<ol>
88+
<li>Set $\alpha \leftarrow 1$.</li>
89+
<li>If $f(x_n - \alpha f'(x_n)/f''(x_n)) \leq f(x_n) - c\,\alpha\,[f'(x_n)]^2/f''(x_n)$, accept $\alpha$.</li>
90+
<li>Otherwise set $\alpha \leftarrow \tau\alpha$ and go to step 2.</li>
91+
</ol>
92+
93+
<p>The plot shows: the function $f(x)$ (blue); the tangent line at $x_n$ (gray dashed); the orange dashed line marking the <em>Armijo RHS threshold</em> $f(x_n) - c\,\alpha\,[f'(x_n)]^2/f''(x_n)$ for the accepted step size. The Armijo condition is satisfied when the function value lies <em>at or below</em> the orange line. Rejected trial steps (from the last iteration) are shown as faded red crosses &#xd7; on the curve; the accepted step is the solid red dot. The vertical black dashed lines mark successive iterates $x_0, x_1, \ldots$</p>
94+
95+
<p>Use the +/&#8722; buttons and sliders to observe how a larger $c$ makes the condition stricter (requiring more backtracking), and how a smaller $\tau$ shrinks the step more aggressively.</p>
96+
</div>
97+
98+
</main>
99+
100+
<script id="simulation_code" type="text/javascript">
101+
class Plot
102+
{
103+
constructor()
104+
{
105+
this.reset();
106+
}
107+
108+
reset()
109+
{
110+
this.num_steps = parseInt(document.getElementById('textInput').value);
111+
this.c = parseFloat(document.getElementById('armijo_c').value);
112+
this.tau = parseFloat(document.getElementById('tau').value);
113+
this.fct = document.getElementById('fct').value;
114+
this.plotFunctions();
115+
}
116+
117+
changeSteps(delta)
118+
{
119+
const el = document.getElementById('textInput');
120+
let val = parseInt(el.value) + delta;
121+
val = Math.max(1, Math.min(30, val));
122+
el.value = val;
123+
this.reset();
124+
}
125+
126+
// ------------------------------------------------------------------ //
127+
// Function definitions
128+
// All four are chosen so that the full Newton step (α=1) fails the
129+
// Armijo condition from the given starting point.
130+
// ------------------------------------------------------------------ //
131+
132+
// f(x) = log(cosh(2x)) — minimum at x=0
133+
// Hessian = 4·sech²(2x) is tiny far from 0 → Newton step is huge.
134+
hyperbolic_function(x) { return Math.log(Math.cosh(2*x)); }
135+
grad_hyperbolic_function(x) { return 2*Math.tanh(2*x); }
136+
hess_hyperbolic_function(x) { const c = Math.cosh(2*x); return 4/(c*c); }
137+
138+
// f(x) = 0.5x² + 2sin(x) — minimum near x ≈ -1.03
139+
// From x=-2.5 the Newton step overshoots to the wrong side.
140+
sinusoidal_function(x) { return 0.5*x*x + 2*Math.sin(x); }
141+
grad_sinusoidal_function(x) { return x + 2*Math.cos(x); }
142+
hess_sinusoidal_function(x) { return 1 - 2*Math.sin(x); }
143+
144+
// f(x) = 0.5x² + 5·exp(−0.5x²) — minima at x ≈ ±1.794
145+
// Gaussian bump makes f non-convex near x=0; the full Newton step
146+
// from x=3 lands in the non-convex region, failing Armijo.
147+
gaussian_bump_function(x) { return 0.5*x*x + 5*Math.exp(-0.5*x*x); }
148+
grad_gaussian_bump_function(x) { return x*(1 - 5*Math.exp(-0.5*x*x)); }
149+
hess_gaussian_bump_function(x) { return 1 + 5*Math.exp(-0.5*x*x)*(x*x - 1); }
150+
151+
// f(x) = x² − 4·log(x²+1) + 1 — minima at x ≈ ±√3, local max at x=0
152+
// Logarithmic term gives a W-shape; Newton step from x=3 overshoots.
153+
log_rational_function(x) { return x*x - 4*Math.log(x*x + 1) + 1; }
154+
grad_log_rational_function(x) { return 2*x*(x*x - 3)/(x*x + 1); }
155+
hess_log_rational_function(x) { const d = x*x + 1; return 2*(x*x-3)/d + 16*x*x/(d*d); }
156+
157+
// ------------------------------------------------------------------ //
158+
// Backtracking line search (Newton direction) — returns accepted α
159+
// ------------------------------------------------------------------ //
160+
backtrack(fct, grad_fct, hess_fct, xn, c, tau)
161+
{
162+
const fn = fct(xn);
163+
const gn = grad_fct(xn);
164+
const hn = hess_fct(xn);
165+
if (hn <= 0) return { alpha: 0, attempts: 0, rejected: [] }; // not a descent direction
166+
const dn = -gn / hn; // Newton direction
167+
let alpha = 1.0;
168+
const maxIter = 60;
169+
const rejected = [];
170+
for (let k = 0; k < maxIter; k++)
171+
{
172+
const xNew = xn + alpha * dn;
173+
// Armijo: f(xn + α·dn) ≤ f(xn) + c·α·f'(xn)·dn
174+
if (fct(xNew) <= fn + c * alpha * gn * dn)
175+
return { alpha: alpha, attempts: k + 1, rejected: rejected, converged: true };
176+
rejected.push(xNew);
177+
alpha *= tau;
178+
}
179+
return { alpha: alpha, attempts: maxIter, rejected: rejected, converged: false };
180+
}
181+
182+
// ------------------------------------------------------------------ //
183+
// Build all Plotly traces
184+
// ------------------------------------------------------------------ //
185+
computeData(fct, grad_fct, hess_fct, x0, data)
186+
{
187+
const c = this.c;
188+
const tau = this.tau;
189+
190+
// ---- main function curve ----
191+
const numPts = 5000;
192+
const xMin = -3, xMax = 3;
193+
const xArr = [], yArr = [];
194+
for (let i = 0; i <= numPts; i++)
195+
{
196+
const xi = xMin + i * (xMax - xMin) / numPts;
197+
xArr.push(xi);
198+
yArr.push(fct(xi));
199+
}
200+
data.push({
201+
x: xArr, y: yArr,
202+
name: "f(x)",
203+
showlegend: true,
204+
line: { color: 'rgb(31, 119, 180)', width: 2 }
205+
});
206+
207+
// ---- collect iterates ----
208+
let iterates = [x0];
209+
let alphas = [];
210+
let attempts = [];
211+
let rejectedAll = [];
212+
let xCur = x0;
213+
let converged = true;
214+
for (let i = 0; i < this.num_steps; i++)
215+
{
216+
const res = this.backtrack(fct, grad_fct, hess_fct, xCur, c, tau);
217+
alphas.push(res.alpha);
218+
attempts.push(res.attempts);
219+
rejectedAll.push(res.rejected);
220+
// Newton update: x_{n+1} = x_n - α * f'(x_n)/f''(x_n)
221+
xCur = xCur - res.alpha * grad_fct(xCur) / hess_fct(xCur);
222+
iterates.push(xCur);
223+
if (!res.converged) { converged = false; break; }
224+
}
225+
226+
// ---- per-step decorations ----
227+
for (let i = 0; i < this.num_steps; i++)
228+
{
229+
const xn = iterates[i];
230+
const xn1 = iterates[i + 1];
231+
const fn = fct(xn);
232+
const gn = grad_fct(xn);
233+
const alpha = alphas[i];
234+
const hn = hess_fct(xn);
235+
const dn = (hn > 0) ? -gn / hn : 0; // Newton direction
236+
const rejPts = rejectedAll[i];
237+
const isLast = (i === this.num_steps - 1);
238+
239+
// Range for the tangent line (centred on xn, wide enough to cover all trials)
240+
const halfW = Math.max(Math.abs(alpha * dn) + 0.5, 0.8);
241+
const lo = xn - halfW;
242+
const hi = xn + halfW;
243+
const nPts = 200;
244+
245+
// Tangent line: f(xn) + f'(xn)*(x - xn)
246+
const txArr = [], tyArr = [];
247+
for (let j = 0; j <= nPts; j++)
248+
{
249+
const xt = lo + j * (hi - lo) / nPts;
250+
txArr.push(xt);
251+
tyArr.push(fn + gn * (xt - xn));
252+
}
253+
data.push({
254+
x: txArr, y: tyArr,
255+
line: { color: 'rgba(100,100,100,0.6)', width: 1.5, dash: 'dash' },
256+
name: "tangent at xₙ",
257+
showlegend: i === 0
258+
});
259+
260+
// Armijo RHS: horizontal dashed line — only for the current (last) iteration
261+
if (isLast)
262+
{
263+
const threshY_val = fn + c * alpha * gn * dn;
264+
data.push({
265+
x: [lo, hi], y: [threshY_val, threshY_val],
266+
mode: 'lines',
267+
line: { color: 'rgba(210,120,0,0.9)', width: 2, dash: 'dash' },
268+
name: "Armijo RHS: f(xₙ) − c·α·[f'(xₙ)]²/f''(xₙ)",
269+
showlegend: true
270+
});
271+
}
272+
273+
// X markers: rejected trial points — only for the last iteration
274+
if (isLast && rejPts.length > 0)
275+
{
276+
data.push({
277+
x: rejPts,
278+
y: rejPts.map(xr => fct(xr)),
279+
mode: 'markers',
280+
marker: { color: 'rgba(180, 60, 60, 0.55)', size: 10, symbol: 'x', line: { width: 2, color: 'rgba(180,60,60,0.55)' } },
281+
name: "rejected step",
282+
showlegend: true
283+
});
284+
}
285+
286+
// Red dot: accepted next point (xn1, f(xn1))
287+
const nBacktracks = attempts[i] - 1;
288+
const stepLabel = nBacktracks === 0
289+
? "accepted (full Newton step)"
290+
: "backtracked " + nBacktracks + "×";
291+
data.push({
292+
x: [xn1], y: [fct(xn1)],
293+
mode: 'markers',
294+
marker: { color: 'red', size: 10 },
295+
name: stepLabel,
296+
showlegend: true
297+
});
298+
299+
// Vertical dashed line at current iterate xn
300+
data.push({
301+
x: [xn, xn], y: [0, fn],
302+
mode: 'lines+markers+text',
303+
line: { color: 'rgb(0,0,0)', width: 2, dash: 'dash' },
304+
text: ["x_" + i, ""],
305+
textposition: "bottom center",
306+
name: "xₙ",
307+
showlegend: i === 0
308+
});
309+
}
310+
311+
// Vertical dashed line for the final iterate
312+
const xFinal = iterates[this.num_steps];
313+
const fFinal = fct(xFinal);
314+
data.push({
315+
x: [xFinal, xFinal], y: [0, fFinal],
316+
mode: 'lines+markers+text',
317+
line: { color: 'rgb(0,0,0)', width: 2, dash: 'dash' },
318+
text: ["x_" + this.num_steps, ""],
319+
textposition: "bottom center",
320+
name: "xₙ",
321+
showlegend: false
322+
});
323+
324+
if (!converged)
325+
{
326+
data.push({
327+
x: [xFinal], y: [fFinal],
328+
mode: 'markers+text',
329+
marker: { color: 'red', size: 10 },
330+
text: ["failed to converge"],
331+
textposition: "top center",
332+
name: "failed to converge",
333+
showlegend: true
334+
});
335+
}
336+
}
337+
338+
plotFunctions()
339+
{
340+
const data = [];
341+
342+
if (this.fct === "Hyperbolic function")
343+
this.computeData(this.hyperbolic_function.bind(this),
344+
this.grad_hyperbolic_function.bind(this),
345+
this.hess_hyperbolic_function.bind(this), 1.5, data);
346+
347+
if (this.fct === "Sinusoidal function")
348+
this.computeData(this.sinusoidal_function.bind(this),
349+
this.grad_sinusoidal_function.bind(this),
350+
this.hess_sinusoidal_function.bind(this), -2.5, data);
351+
352+
if (this.fct === "Gaussian bump function")
353+
this.computeData(this.gaussian_bump_function.bind(this),
354+
this.grad_gaussian_bump_function.bind(this),
355+
this.hess_gaussian_bump_function.bind(this), 3.0, data);
356+
357+
if (this.fct === "Log-rational function")
358+
this.computeData(this.log_rational_function.bind(this),
359+
this.grad_log_rational_function.bind(this),
360+
this.hess_log_rational_function.bind(this), 3.0, data);
361+
362+
const layout = {
363+
title: "Newton's method with backtracking line search (Armijo condition)",
364+
autosize: true,
365+
xaxis: { range: [-4, 4] },
366+
yaxis: { range: [-4, 8] }
367+
};
368+
369+
Plotly.newPlot('plotOutput', data, layout);
370+
}
371+
}
372+
373+
const plot = new Plot();
374+
plot.reset();
375+
</script>
376+
377+
</body>
378+
</html>

‎index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ I hope the examples and their documentation help you to learn something about th
1717
* [Newton's method](examples/Newton_solver.html)
1818
* [Newton's method for minimization (1D)](examples/Newton_minimization.html)
1919
* [Newton's method for minimization (2D)](examples/Newton_minimization_2D.html)
20+
* [Newton's method with backtracking line search](examples/Newton_with_backtracking_line_search.html)
2021
* [Newton's Method — The Saddle Point Problem](examples/Newton_saddle_3D.html)
2122

2223
# Particles

0 commit comments

Comments
 (0)