# Optimization

This unit covers the following ideas. In preparation for the quiz and exam, make sure you have a lesson plan containing examples that explain and illustrate the following concepts.
1. Explain the properties of the gradient, its relation to level curves and level surfaces, and how it can be used to find directional derivatives.
2. Find equations of tangent planes using the gradient and level surfaces. Use the derivative (tangent planes) to approximate functions, and use this in real world application problems.
3. Explain the second derivative test in terms of eigenvalues. Use the second derivative test to optimize functions of several variables.
4. Use Lagrange multipliers to optimize a function subject to constraints. %Explain why Lagrange multipliers works by considering the gradients of the function and the constraint at a maximum.
You'll have a chance to teach your examples to your peers prior to the exam. The following homework problems line up with the topics we will discuss in class. {\noindent \footnotesize
{|l|c|l|l|l|l|}\hline Topic (11th ed.) &Sec &Basic Practice &Good Problems &Thy/App &Comp \\\hline Directional Derivatives and the Gradient & 14.5&1-22 &23-32 &33-36 & \\\hline Tangent Planes and approximation & 14.6&1-22 &23-24, 47-58, 60-63 & 59& \\\hline 2nd Derivative Test (use eigenvalues) & 14.7&1-38 &39-44, 49-52, &45-48, 53-64 &65-70 \\\hline Lagrange Multipliers & 14.8&1-32 &33-40 &41-44 &45-50 \\\hline
%Taylor Polynomials & 14.10& & & & \\\hline
} {\noindent \footnotesize
{|l|c|l|l|l|l|}\hline Topic (12th ed.) &Sec &Basic Practice &Good Problems &Thy/App &Comp \\\hline Directional Derivatives and the Gradient & 14.5&1-24 &25-36 &37-40 & \\\hline Tangent Planes and approximation & 14.6&1-22 &23-24, 31-32, 49-62, 64-67 &63 & \\\hline 2nd Derivative Test (use eigenvalues) & 14.7&1-38 &39-44, 49-60, &45-48, 61-68 &69-74 \\\hline Lagrange Multipliers & 14.8&1-32 &33-40 &41-44 &45-50 \\\hline
%Taylor Polynomials & 14.10& & & & \\\hline
}

See Larson 13.6–13.7.

The derivative $Df$ of a function $f:\mathbb{R}^n\to\mathbb{R}$ (one output dimension) is called the gradient of $f$, and written $\vec \nabla f$, when we want to emphasize that the derivative is a vector field.

You'll want a computer to help you construct the graphs, particularly $h$. Please use the Mathematica introduction in Brainhoney. You could use Wolfram Alpha (use the links in the function chapter if you forgot how to graph).
Consider the functions $f(x,y)=9-x^2-y^2$, $g(x,y)=2x-y$, and $h(x,y)=\sin x\cos y$.
1. See Sage. You can modify these commands to help in the plots below too.
Compute $\vec \nabla f(x,y)$. Then draw both $\vec \nabla f$ and several level curves of $f$ on the same axes.
2. Compute $\vec \nabla g(x,y)$. Then draw both $\vec \nabla g$ and several level curves of $g$ on the same axes.
3. Compute $\vec \nabla h(x,y)$. Then draw both $\vec \nabla h$ and several level curves of $h$ on the same axes.
4. What relationships do you see between the gradient vector field and level curves?
When you present in class, be prepared to provide rough sketches of the level curves and gradients of each function.

The next problem investigates why the gradient vector and the level curve have the relationship you hopefully saw.

Suppose $\vec r(t)$ is a level curve of $f(x,y)$.
1. Suppose you know that at $t=0$, the value of $f$ at $\vec r(0)$ is 7. What is the value of $f$ at $\vec r(1)$? [What does it mean to be on a level curve?]
2. As you move along the level curve $\vec r$, how much does $f$ change? Use this to tell the class what $\ds\frac{df}{dt}$ must equal.
3. At points along the level curve $\vec r$, we have the composite function $f(\vec r(t))$. Compute the derivative $\ds\frac{df}{dt}$ using the chain rule.
4. Use your work from the previous parts to explain why the gradient always meets the level curve at a 90$^\circ$ angle. We say that the gradient is normal to level curves (i.e., a gradient vector is orthogonal to the tangent vector of the curve).

In the derivative chapter, we extended differential notation from $dy=f' dx$ to $d\vec y = D\vec f d\vec x$. The key idea is that a small change in the output variables is approximated by the product of the derivative and a small change in the input variables. As a quick refresher, if we have the function $z=f(x,y)$, then differential notation states that $$dz = \begin{bmatrix}f_x&f_y\end{bmatrix} \begin{bmatrix}dx\\dy\end{bmatrix}.$$

Suppose the temperature at a point in the plane is given by the function $T(x,y)=x^2-xy-y^2$ degrees Fahrenheit. A particle is at $P=(2,3)$.
1. Use differentials to estimate the change in temperature if the particle moves 1 unit in the direction of $\vec u=\left(3,4\right)$. [Hint: Find a unit vector in that direction.]
2. What is the actual change in temperature if the particle moves 1 unit in the direction of $\vec u=\left(3,4\right)$?
3. Use differentials to estimate the change in temperature if the particle moves about .2 units in the direction of $\vec u=\left(3,4\right)$.

Let's generalize the calculation we just did into a formula for any $f(x,y)$.

Suppose that $z=f(x,y)$ is a differentiable function (so the derivative is the matrix $\begin{bmatrix}f_x&f_y\end{bmatrix}$). Remember to use differential notation in this problem.
1. If $(dx,dy)=(1,0)$, which means we've moved one unit in the $x$ direction while holding $y$ constant, what is $dz$?
2. If $(dx,dy)=(0,1)$, which means we've moved one unit in the $y$ direction while holding $x$ constant, what is $dz$?
3. Consider the direction $\vec u=(2,3)$. Find a unit vector in the direction of $\vec u$. If we move one unit in the direction of $\vec u$, what is $dz$? [It's all right to leave you answer as a dot product.]
The directional derivative of $f$ in the direction of the unit vector $\vec u$ at a point $P$ is defined to be $$D_{\vec u} f(P) = Df(P) \vec u = \vec \nabla f \cdot \vec u.$$ We dot the gradient of $f$ with the direction vector $\vec u$. The partial derivative of $f$ with respect to $x$ is precisely the directional derivative of $f$ in the $(1,0)$ direction. Similarly, the partial derivative of $f$ with respect to $y$ is precisely the directional derivative of $f$ in the $(0,1)$ direction. This definition extends to higher dimensions.

Note that in the definition above, we require the vector $\vec u$ to be a unit vector. If you are asked to find a directional derivative in some direction, make sure you start by finding a unit vector in that direction. We want to deal with unit vectors because when we say something has a slope of $m$ units, we want to say “The function rises $m$ units if we run $1$ unit.”

Consider the function $f(x,y) = 9-x^2-y^2$.
1. Draw several level curves of $f$.
2. At the point $P=(2,1)$, place a dot on your graph. Then draw a unit vector based at $P$ that points in the direction $\vec u=(3,4)$ [not to the point $(3,4)$, but in the direction $\vec u=(3,4)$]. If you were to move in the direction $(3,4)$, starting from the point $(2,1)$, would the value of $f$ increase or decrease?
3. Find the slope of $f$ at $P=(2,1)$ in the direction $\vec u=(3,4)$ by finding the directional derivative. This should agree with your previous answer.
4. If you stand at $Q=(-2,3)$ and move in the direction $\vec v= (1,-1)$, will $f$ increase or decrease? Find the directional derivative of $f$ in the direction $\vec v=(1,-1)$ at the point $Q=(-2,3)$.
Recall that the directional derivative of $\vec f$ in the direction $\vec u$ is the dot product $\vec \nabla f\cdot \vec u = |\vec \nabla f| |\vec u|\cos\theta$. In this problem, you'll explain why the gradient points in the direction of greatest increase.
• What is the angle between the two vectors $\vec \nabla f$ and $\vec u$? [Hint: we learned a formula for the angle between two vectors back in the vectors chapter.]
1. Why is the directional derivative of $\vec f$ the largest when $\vec u$ points in the exact same direction as $\vec \nabla f$? [Hint: What angle maximizes the cosine function?]
2. When $\vec u$ points in the same direction as $\vec \nabla f$, show that $D_{\vec u}f = |\vec \nabla f|$. In other words, explain why the length of the gradient is precisely the slope of $f$ in the direction of greatest increase (the slope in the steepest direction).
3. Which direction points in the direction of greatest decrease?
Suppose you are looking at a topographical map (see Wikipedia for an example). On this topographical map, each contour line represents 100 ft in elevation. You notice in one section of the map that the contour lines are really close together, and they start to form circles around a spot on the graph. You notice in another section of the map that the contour lines are spaced quite far apart. Let $f(x,y)$ be the elevation of the land, so that the topographical map is just a contour plot of $f$.
1. For example, you can look at a contour plot of $f(x,y) = (x+1)^3-3(x+1)^2-y^2+2$ in Sage.
Where is the slope of the terrain larger, in the section with closely packed contour lines, or the section with contour lines that are spread out. In which section will the gradient be a longer vector?
2. At the very top of a mountain, or the very bottom of a valley, will the gradient be a long vector or a short vector?

Summarizing our results, we have the following theorem.

Let $f$ be a continuously differentiable function, with $\vec r$ a level curve of the function.
• The gradient is always normal to level curves, meaning $\vec \nabla f\cdot \dfrac{d\vec r}{dt}=0$.
• The gradient points in the direction of greatest increase.
• The directional derivative of $f$ in the direction of the gradient is the length of the gradient. Symbolically, we write $D_{\vec \nabla f}f = |\vec \nabla f|$.
• At a maximum or minimum, the gradient is the zero vector.

The next few problems have you practice using differentials, and then obtain tangent lines and planes to curves and surfaces using differentials.

See Sage.
Consider the function $f(x,y)=x^2+y^2$. Consider the level curve $C$ given by $f(x,y)=25$. Our goal is to find an equation of the tangent line to $C$ at $P=(3,-4)$.
1. Draw $C$. Compute $\vec \nabla f$ and add to your graph the vector $\vec \nabla f(P)$.
2. We know the point $P=(3,-4)$ is on the tangent line. Let $Q=(x,y)$ represent another point on the tangent line. Add to your graph the point $Q$ and the vector $\vec {PQ} = (x-3,y+4)$.
3. Why are $\vec \nabla f(P)$ and $\vec{PQ}$ orthogonal? Use this fact to write an equation of the tangent line.
4. What is a normal vector to the line?
This problem might be better if on part 3 we wrote, “Why does the dot product $\vec \nabla f(P)\cdot\vec{PQ}$ equal zero? Compute this dot product. Why is this an equation of the tangent line?” To many students are correctly giving a vector equation of the tangent line, by reversing the order of the gradient, negating one of the terms, and then using (x,y)=dir vec(t)+point.

The previous problem had you give an equation of the tangent line to a level curve, by using differential notation. The next problems asks you to repeat this idea and give an equation of a tangent plane to a level surface.

See Sage.
Consider the function $f(x,y,z)=x^2+y^2+z^2$. Consider the level surface $S$ given by $f(x,y,z)=9$. Our goal is to find an equation of the tangent plane to $S$ at $P=(1,2,-2)$.
1. Draw $S$.
2. Compute $\vec \nabla f$. Add to your graph the vector $\vec \nabla f(P)$, with its base at $P$.
3. We know the point $P=(1,2,-2)$ is on the tangent plane. Let $Q=(x,y,z)$ be any other point on the tangent plane. What is the component form of the vector $\vec {PQ}$?
4. Why are $\vec \nabla f(P)$ and $\vec{PQ}$ orthogonal? Use this fact to write an equation of the tangent plane.
5. What is a normal vector to the plane?
Find an equation of the tangent plane to the hyperboloid of one sheet $1=x^2-y^2+z^2$ at the point $(-3,3,1)$.
%Also give an equation of the normal line (the line that sticks straight up out of the surface). FYI, the normal line is used in computer graphics to know how to shade an object.
The two surfaces $x^2+y^2+z^2=14$ and $3x+4y-z=-1$ intersect in a curve $C$. Draw both surfaces, and show us the curve $C$. Then, at the point $(2,-1,3)$, find an equation of the tangent line to this curve. [Hint: The line is in both tangent planes, so it is orthogonal to both normal vectors. The cross product gets you a vector that is orthogonal to two vectors.]
%
This is almost exactly like TODO and feels redundant. NOTE: I added this problem because it is a pet favorite of mine. I decided to remove it. I may add it back next semester as an optional problem. It's not exactly easy to see the answer, whereas it is simple to see the solution without differentials on the volume problem.
% Consider an electrical circuit with two parallel resistors, each having resistance {$R_1$} and {$R_2$}. The total resistance $R$ in this circuit is given by $$\frac{1}{R} = \frac{1}{R_1}+\frac{1}{R_2}.$$ %
1. %
2. Show that $DR = \begin{bmatrix}\frac{R^2}{R_1^2} & \frac{R^2}{R_2^2} \end{bmatrix}$. Then compute $dR$. %
3. When you purchase a resistor, it comes with a stated resistance. However, manufacturing defects always occur, and the stated resistance on each resistor is not the actual resistance. This is where differentials come in handy. % If {$R_1$} changes from an expected 10 ohms to an actual 9.9 ohms, and {$R_2$} changes from 20 to 20.2, would you expect a positive or negative change in the total resistance {$R$}? Use differentials to make your claim, and then back it up by computing the total resistance using (10,20) and then using $(9.9,20.2)$. %
4. Suppose both $R_1$ and $R_2$ have the same resistance, and each have a tolerance of $dR$. Using differentials, what would you expect as the maximum possible error for the total resistance. You should show that running resistors in parallel reduces the error. (For those of you familiar with electrical systems, running resistors in series causes the errors to add.) %
%