MA 574 — Numerical Optimization: Introduction

MA 574 — Numerical Optimization, Fall 2026

Instructor: Xiangxiong Zhang (feel free to call me X)
Course webpage: www.math.purdue.edu/~zhan1966/teaching/574/2026/
Syllabus: on Brightspace — please read it carefully.
Office hours: Wednesday 3:30–5:30pm, or by appointment.

Contents of these slides:

Scope of this semester.
Some critical points in the syllabus.

Plan for this semester

There are many different types of optimization problems, but we mainly focus on the convergence of algorithms minimizing a convex function $f(x)$ with a large scale:

Convex problems

Part I: some classical algorithms for minimizing a smooth function $f(x)$, such as gradient descent, accelerated gradient descent, Newton's method, etc.
Part II: algorithms for composite optimization of minimizing $f(x)+g(x)$ where functions are convex, but at least one of them is not differentiable, e.g., $f(x)=\|x\|_1$.
Part III: stochastic-type algorithms, such as stochastic gradient descent.

Nonconvex problems

Part IV: minimization over a Riemannian manifold constraint.
Part V: if time permits, convergence of gradient descent for nonconvex problems.

Differences compared to other courses on campus

There are other grad optimization courses offered in CS and engineering at Purdue.
MA 574 is the only math graduate course on numerical optimization. Fall 2026 will be the second time it is offered, covering convergence theory of
1. Smooth optimization methods such as (accelerate) gradient descent.
2. Nonsmooth convex optimization such as proximal gradient and splitting methods.
3. Randomized and stochastic methods.
4. Riemannian optimization.
At most 20%~30% of MA 574 are classical topics covered in a standard textbook/course:
- Many methods and techniques such as Nesterov's acceleration, stochastic gradient descent, and Riemannian optimization became popular only after 2000, so they were usually not covered in a book/course 20 (or even 10) years ago.
- Contents such as Riemannian optimization are not covered in other courses.

Examples

Part I: for $\min\limits_x f(x)$, the gradient descent method is \[x_{k+1}=x_k-\eta_k \nabla f(x_k)\]
When and why does gradient descent converge? How fast does it converge?
Prerequisites for Part I:
- Calculus: gradient, Hessian, Taylor's Theorem, …
- Linear algebra: eigenvalues, singular values, etc.
Part II: we will introduce subderivatives, the proximal operator, and algorithms using the subderivatives. We will use monotonicity of operators to prove convergence. \[\min \|x\|_1+\|Ax-b\|_2^2\] We will need some knowledge of convex non-differentiable functions, which will be covered in class.

Example: Part II — image denoising

Part II: here is another example of nonsmooth convex optimization, for denoising a given noisy image $A$ via TV (total variation) norm minimization \[ \min_x \|x\|_{TV}+\lambda\|x-A\|_2^2,\] where $\|x\|_{TV}=\sum\limits_{i,j} \sqrt{|x_{i,j}-x_{i+1, j}|^2+|x_{i,j}-x_{i, j+1}|^2}$.

Noisy input (left) and the result after 150 iterations of Gprox-PDHG (equivalent to ADMM) with $\lambda=20$ (right).

The algorithm ADMM will be covered in Part II.
Large scale means: if the dimension of $x$ is $n$, then only $\mathcal{O}(n)$ storage is acceptable. What is $n^2$?

Examples

Part III: for minimizing $f(x):=\sum\limits_{i=1}^N f_i(x)$, the full gradient is $\nabla f(x)=\sum\limits_{i=1}^N \nabla f_i(x)$. We can use the stochastic gradient like \[\nabla_S f(x):=\sum\limits_{i\in S} \nabla f_i(x)\] where $S$ is a random small subset of $\{1,2,\cdots,N\}$. The stochastic gradient descent can be defined as: \[ x_{k+1}=x_k-\eta_k \nabla_{S_k} f(x_k).\] In order to analyze the convergence, we need some probability knowledge, which will be introduced.

An example where $N$ is too large: recommendation systems for customers rating products (movies, merchandise, etc.).

Examples

Part IV: consider minimizing $f(x)$ with $x\in\mathcal{M}\subset \mathbb{R}^N$ where $\mathcal{M}$ is a Riemannian manifold. If you have not heard of manifolds, just think of $\mathcal{M}$ as a surface, e.g., a unit sphere.

Diagram of two coordinate charts on a manifold. A curved surface at the bottom contains two overlapping dashed regions labeled U and V. An arrow phi maps U up to a chart phi(U) in a copy of R^d at upper left; an arrow psi maps V up to a chart psi(V) in another copy of R^d at upper right. The overlap is shown in both charts as phi(U intersect V) and psi(U intersect V), and two horizontal arrows between the charts indicate the transition maps psi composed with phi inverse, and phi composed with psi inverse. — (a) Charts: two overlapping coordinate charts and their transition maps.

Diagram of a saddle-shaped manifold M with a flat plane, the tangent space T_x M, touching it at a point x. From x, a tangent vector xi points along the plane; a curved arrow leads from the tip of xi down onto the surface to a point labeled R_x(xi), illustrating a retraction mapping the tangent vector back onto the manifold. — (b) Tangent space and retraction.

Manifold over $\mathbb{R}$: $M$ is a set that is locally diffeomorphic to $\mathbb{R}^d$.
Tangent space: a tangent vector is tangent to a curve on $M$.
For $f(X)$ defined on $M$, the Riemannian gradient $\operatorname{grad} f(X)$ is a tangent vector.

An example of Riemannian gradient

Consider $\min\limits_{X\in\mathcal{M}} f(X)=\tfrac12 \|\mathcal{A}(X)-b\|^2$ where $\mathcal{A}$ is a linear operator and $\mathcal{M}$ is an embedded manifold in $\mathbb{R}^N$.
Gradient: $\nabla f(X)=\mathcal{A}^*(\mathcal{A}(X)-b)$.
The Riemannian gradient is the projection of $\dfrac{\partial f(X)}{\partial X}=\mathcal{A}^*(\mathcal{A}(X)-b)$ onto $T_X \mathcal{M}$.

Hand-drawn sketch of a dome-shaped manifold M = S_+^{n,k}, a subset of R^{n by n}, with a flat tangent space T_x M drawn through a point x_k = X on the surface. A blue arrow rising out of the plane is labeled with the Euclidean gradient, partial f over partial X equals A-star of (A(X) minus b). A red arrow lying in the tangent plane is its projection, P_{T_x M} of the gradient. A dashed purple arrow drops from the tip of that projection back down onto the surface to the next iterate x_{k+1}. — The Riemannian gradient (red) is the Euclidean gradient (blue) projected onto the tangent space $T_X\mathcal{M}$; the next iterate $x_{k+1}$ is retracted back onto the manifold.

Focuses of this course

We focus on the analysis of practical algorithms, i.e., why they converge.
Applications/implementation will be barely mentioned.
A final presentation/report (depending on our schedule) is required, by reading a paper and/or implementing some classical/novel algorithms. Examples of possible choices of papers:
- Muon: matrix orthogonalization for neural-network optimization.
- practical stochastic second-order optimization.
- Riemannian Adam/AdaGrad for low-rank NN weights.

Goals/outcomes of this course

Before 2025, by the end of this semester I would have expected you to:
- Be familiar with the major arguments used in convergence proofs.
- Be comfortable reading and carrying out a convergence analysis.
- Know more about both analysis and optimization methods.
Coding and applications are important, but we will not have time for them in class.

2026 is a landmark year of AI: AI agents are now heavily used not only for industrial coding but also for academic research. Frontier models ace math olympiads, settle open conjectures, etc.
You can bet that AI knows everything about MA 574.
In Fall 2026 the learning goals are exactly the same — without them, we would be left with doubts:
- Is an AI-generated analysis or proof actually correct? How would we know?
- Are there better ways to prompt an LLM? Why have other people used AI to do impressive things, but we cannot?

Using AI in MA 574

Purdue now has an AI competency graduation requirement.
In this course, I encourage you to use AI to assist learning (but not cheating…). Coding without AI is on its way to becoming what punching cards was in 1964.

Programming in 1964: punching cards.

This kind of “supercomputer” was not fully retired in US universities until the early 1980s.
(I think) AI competency has three layers (take MA 574 as the example):
1. Be comfortable using AI agent systems. Try a CLI (command line interface) agent like Codex or Claude Code: a chatbot (ChatGPT, Claude) can only talk about code, while an agent writes it, runs it, and fixes its own errors.
2. Know the basic stuff — i.e., the goals of MA 574: the convergence analysis itself, and whether a proof is actually correct.
3. Know how to use AI for optimization productively and efficiently. You have to try it out and learn along the way, but this is impossible without (a) or (b).

Importance of traditional classroom learning in AI era

View from the driver's seat of a Tesla on a two-lane highway at sunset. Nobody's hands are on the steering wheel in the foreground. The center touchscreen shows Self-Driving engaged at 75 mph, with the car rendered centered in its lane and surrounding traffic displayed ahead. — Tesla FSD (Supervised) does the heavy work on a road trip — but you still supervise it the whole time.

Screenshot of an NBC News Now segment titled The Future of Everything. Students carrying backpacks gather around a table beneath a tall banner reading Claude. The on-screen headline reads: A.I. hackathon shows humans are still critical to software coding. A description panel below reads: Georgia Tech get three hours to build an app using Claude AI. — NBC News, May 7, 2026: humans are still critical.

Tesla FSD (Supervised) handles 99% of driving conditions. For the other 1%, someone must still be behind the wheel — with at least a driver's license.
As a job applicant, you compete with someone who can use AI well but not AI.
And you can use AI well only on what you already know. Ask it for MATLAB code, AI gives one and tells you MATLAB is slow. You get faster code only if prompting AI to:
1. vectorize loops whenever possible;
2. wrap whatever cannot be vectorized in C++ via mex.
So you still need to learn math and coding, even if AI can do it all: only the knowledgeable can use AI well enough to stay competitive.
On the other hand, do use AI as your free teaching assistant — to digest hard concepts and pick up new math/coding tricks.

Syllabus: grading and attendance

Grades and grade reports

Attendance: 60%
In-class quiz: 10% (prove one given theorem; date is Oct 16 Friday)
Written report: 15% (a list of papers for reading will be given later)
Presentation: 15% (12 people present in class; others submit video recordings)

Attendance policy

Attendance is required and checked on a regular basis.
The first 3 unexcused absences cost you nothing. From the fourth on, each one deducts 5% of the score: 4 absences → −5%, 5 → −10%, 6 → −15%, and so on.
Examples of excused absences: illness, bereavement, and conference travel. Except in emergencies, tell me in advance and provide documentation — a physician's note, a letter from the Office of the Dean of Students, or an email from research advisor.

Syllabus: academic integrity and AI policy

Intellectual property. Lecture notes, Zoom videos, and typed-up notes are all copyrighted. Uploading any of these to any forum or website is strictly prohibited.
Academic integrity is one of the highest values that Purdue University holds. You could, of course, sign the attendance sheet or write a report or slides for someone else — but “all incidents of academic misconduct will be forwarded to OSRR, where university penalties, including removal from the university, may be considered.”
AI policy.
1. Definitely use AI to assist your learning, reading, and writing of the report.
2. A report or slides generated by one prompt — or a few prompts with minimal human input — will be penalized. Use AI as a tool, not as a replacement for yourself.
3. The final report must include a required section explaining exactly how you used AI in writing it.

My 2 cents for using AI

Two horizontal chevron pipelines. The top one, labeled Without AI, has three equal stages: Idea, Implementation, Validation. The bottom one, labeled With AI, has the same three stages, but the Idea chevron is much wider while the Implementation chevron has shrunk to a small fraction of its former width. — AI shrinks implementation — idea takes up space.

A four-box cycle titled Positive AI Learning Feedback Loop. Arrows run clockwise: Use AI Better leads to Faster Learner, which leads to Deeper / Better Knowledge, which leads to Ask Better Questions to AI, which leads back to Use AI Better. — Learning and using AI reinforce each other.

Tools keep changing — typewriters and postal mail in the 1980s, email and LaTeX in the 1990s, Google and GPUs in the 2010s, AI agents in the 2020s — but the workflow never changes: Idea → Implementation → Validation.
The faster implementation becomes, the more important ideas are.
Better learning (classroom, books, seminars, conferences) ⇒ better ideas. So in the AI era, learning matters more, not less.
Never AI everything but do use AI to assist your learning.
Rigorous STEM courses are supposed to train critical thinking, which matters more than ever in the AI era. Mastering the convergence analysis of numerical optimization algorithms is one way to sharpen it.