$$

Differential calculus in higher dimension

WARNING

The information in this section is being updated.

In this part of the course we work on the following skills:

• Become comfortable working with coordinates in arbitrary dimension.
• Develop an intuition for working with vector fields.
• Understand the subtleties of derivatives in dimension greater than 1, evaluate and manipulate partial derivatives, directional derivatives, Jacobian.

See also the exercises associated to this part of the course.

Here we start to consider higher dimensional space. That is, instead of $\mathbb{R}$ we consider ${\mathbb{R}}^n$ for $n\in \mathbb{N}$. We will particularly focus on 2D and 3D but everything also holds in any dimension. Going beyond $\mathbb{R}$ we have more options for functions and correspondingly more options for derivatives. Various different notation is commonly used. Here we will primarily use $(x,y)\in {\mathbb{R}}^2$, $(x,y,z)\in {\mathbb{R}}^3$ or, more generally, $\mathbf{x}=(x_1,x_2,\dots ,x_n)\in {\mathbb{R}}^n$ where $x_1\in \mathbb{R},\dots ,x_n\in \mathbb{R}$. For example, ${\mathbb{R}}^2$ is the plane, ${\mathbb{R}}^3$ is 3D space.

Definition (inner product)

$$\mathbf{x}\cdot \mathbf{y}=\sum _{k=1}^n{x}_k{y}_k\in \mathbb{R}$$

We recall that the inner product being zero has a geometric meaning, it means that the two vectors are orthogonal. We also recall that the "length" of a vector is given by the norm, defined as follows.

Definition (norm)

$\Vert \mathbf{x}\Vert =\sqrt{\mathbf{x}\cdot \mathbf{x}}={(\sum _{k=1}^n{x}_k^2)}^{\frac{1}{2}}$.

For example, in ${\mathbb{R}}^2$ then $\Vert (x,y)\Vert =\sqrt{x^2+y^2}$. There are various convenient properties for working with norms and inner products, in particular, the Cauchy-Schwarz inequality $|x\cdot y|\le \Vert \mathbf{x}\Vert \Vert \mathbf{y}\Vert$ and the triangle inequality $\Vert \mathbf{x}+\mathbf{y}\Vert \le \Vert \mathbf{x}\Vert +\Vert \mathbf{y}\Vert$.

The primary higher-dimensional functions we consider in this course are:

• Scalar fields: $f:{\mathbb{R}}^n\to \mathbb{R}$
• Vector fields: $\mathbf{F}:{\mathbb{R}}^n\to {\mathbb{R}}^n$
• Paths: $\mathit{\alpha }:\mathbb{R}\to {\mathbb{R}}^n$
• Change of coordinates: $\mathbf{x}:{\mathbb{R}}^n\to {\mathbb{R}}^n$

These possibilities all fit into the general pattern of $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$ for $n,m\in \mathbb{N}$ but tradition and use of the function gives us different terminology and symbols. Such functions are useful for representing various practical things, for example: gravitational force; temperature in a region; wind velocity; fluid flow; electric field; etc.

Open sets, closed sets, boundary, continuity

Let $\mathbf{a}\in {\mathbb{R}}^n$, $r>0$. The open $n$-ball of radius $r$ and centre $\mathbf{a}$ is written as

$$B(\mathbf{a},r):=\{\mathbf{x}\in {\mathbb{R}}^n:\Vert \mathbf{x}-\mathbf{a}\Vert <r\}.$$

Definition (interior point)

Let $S\subset {\mathbb{R}}^n$. A point $\mathbf{a}\in S$ is said to be an interior point if there is $r>0$ such that $B(\mathbf{a},r)\subset S$. The set of all interior points of $S$ is denoted $\mathrm{int}S$.

Definition (open set)

A set $S\subset {\mathbb{R}}^n$ is said to be open if all of its points are interior points, i.e., if $\mathrm{int}S=S$.

Interior points are the centre of a ball contained within the set

For example, open intervals, open disks, open balls, unions of open intervals, etc., are all open sets.

Lemma

Let $r>0$, $\mathbf{a}\in {\mathbb{R}}^n$. The set $B(\mathbf{a},r)\subset {\mathbb{R}}^n$ is open.

Proof

Let $\mathbf{b}\in B(\mathbf{a},r)$. It suffices to show that $\mathbf{b}$ is an interior point. (1) Let $r_1=\parallel \mathbf{b}-\mathbf{a}\parallel <r$. (2) Let $r_2=(r-r_1)/2$. (3) We claim that $B(\mathbf{b},r_2)\subset B(\mathbf{a},r)$: In order to see this take any $\mathbf{c}\in B(\mathbf{b},r_2)$ and observe that

$$\parallel \mathbf{c}-\mathbf{a}\parallel \le \parallel \mathbf{c}-\mathbf{b}\parallel +\parallel \mathbf{b}-\mathbf{a}\parallel \le r_2+r_1=\frac{r+r_1}{2}<r.$$

Observe that the radius of the ball will be small for points close to the boundary.

Definition (Cartesian product)

If $A_1\subset \mathbb{R}$, $A_2\subset \mathbb{R}$ then the Cartesian product is defined as

$$A_1\times A_2:=\{(x,y):x\in A_1,y\in A_2\}\subset {\mathbb{R}}^2.$$

Analogously the Cartesian product can be defined in higher dimensions: If $A_1\subset {\mathbb{R}}^m$, $A_2\subset {\mathbb{R}}^n$ then the Cartesian product $A_1\times A_2$ is defined as the set of all points $(x_1,\dots ,x_m,y_1,\dots ,y_n)\in {\mathbb{R}}^{m+n}$ such that $(x_1,\dots ,x_m)\in A_1$ and $(y_1,\dots ,y_n)\in A_2$.

Lemma

If $A_1,A_2$ are open subsets of $\mathbb{R}$ then $A_1\times A_2$ is an open subset of ${\mathbb{R}}^2$.

Proof

Let $\mathbf{a}=(a_1,a_2)\in A_1\times A_2\subset {\mathbb{R}}^2$. Since $A_1$ is open there exists $r_1>0$ such that $B(a_1,r_1)\subset A_1$. Similarly for $A_2$. Let $r=min\{r_1,r_2\}$. This all means that $B(\mathbf{a},r)\subset B(a_1,r_1)\times B(a_2,r_2)\subset A_1\times A_2$.

If $A_1,A_2$ are intervals then $A_1\times A_2$ is a rectangle

Discussing the "interior" of the set naturally suggests the topic of the "boundary" of the set. In the following definitions we develop this idea.

Definition (exterior points)

Let $S\subset {\mathbb{R}}^n$. A point $\mathbf{a}\notin S$ is said to be an exterior point if there exists $r>0$ such that $B(\mathbf{a},r)\cap S=\mathrm{\varnothing }$. The set of all exterior points of $S$ is denoted $\mathrm{ext}S$.

Observe that $\mathrm{ext}S$ is an open set. We use the notation $S^c={\mathbb{R}}^n\setminus S$ and we say that $C^c$ is the complement of the set $S$.

Definition (boundary)

The set ${\mathbb{R}}^n\setminus (\mathrm{int}S\cup \mathrm{ext}S)$ is called the boundary of $S\subset {\mathbb{R}}^n$ and is denoted $\partial S$.

Definition (closed)

A set $S\subset {\mathbb{R}}^n$ is said to be closed if $\partial S\subset S$.

Lemma

$S$ is open $⟺$ $S^c$ is closed.

Proof

Observe that ${\mathbb{R}}^n=\mathrm{int}S\cup \partial S\cup \mathrm{ext}S$ (disjointly). If $\mathbf{x}\in \partial S$ then, for every $r>0$, $B(\mathbf{x},r)\cap S\ne \mathrm{\varnothing }$ and so $\mathbf{x}\in \partial (S^c)$. Similarly with $S$ and $S^c$ swapped and so $\partial S=\partial (S^c)$. If $S$ is open then $\mathrm{int}S=S$ and $S^c=\mathrm{ext}S\cup \partial S=\mathrm{ext}S\cup \partial (S^c)$ and so $S^c$ is closed. If $S$ is not open then there exists $\mathbf{a}\in \partial S\cap S$. Additionally $\mathbf{a}\in \partial (S^c)\cap S$ hence $S^c$ is not closed.

Limits and continuity

Let $S\subset {\mathbb{R}}^n$ and $\mathbf{f}:S\to {\mathbb{R}}^m$. If $\mathbf{a}\in {\mathbb{R}}^n$, $\mathbf{b}\in {\mathbb{R}}^m$ we write $\underset{\mathbf{x}\to \mathbf{a}}{lim}\mathbf{f}(\mathbf{x})=\mathbf{b}$ to mean that $\Vert \mathbf{f}(\mathbf{x})-\mathbf{b}\Vert \to 0$ as $\Vert \mathbf{x}-\mathbf{a}\Vert \to 0$. Observe how, if $n=m=1$, this is the familiar notion of continuity for functions on $\mathbb{R}$.

Definition (Continuous)

A function $\mathbf{f}$ is said to be continuous at $\mathbf{a}$ if $\mathbf{f}$ is defined at $\mathbf{a}$ and $\underset{\mathbf{x}\to \mathbf{a}}{lim}\mathbf{f}(\mathbf{x})=\mathbf{f}(\mathbf{a})$. We say $\mathbf{f}$ is continuous on $S$ if $\mathbf{f}$ is continuous at each point of $S$.

Even functions which look "nice" can fail to be continuous as we can see in the following example.

Example (continuity in higher dimensions)

Let $f$ be defined, for $(x,y)\ne (0,0)$, as

$$f(x,y)=\frac{xy}{x^2+y^2}$$

and $f(0,0)=0$. What is the behaviour of $f$ when approaching $(0,0)$ along the following lines?

line	value
$\{x=0\}$	$f(0,t)=0$
$\{y=0\}$	$f(t,0)=0$
$\{x=y\}$	$f(t,t)=\frac{1}{2}$
$\{x=-y\}$	$f(t,t)=-\frac{1}{2}$

Theorem

Suppose that $\underset{\mathbf{x}\to \mathbf{a}}{lim}\mathbf{f}(\mathbf{x})=\mathbf{b}$ and $\underset{\mathbf{x}\to \mathbf{a}}{lim}\mathbf{g}(\mathbf{x})=\mathbf{c}$. Then

1. $\underset{\mathbf{x}\to \mathbf{a}}{lim}(\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x}))=\mathbf{b}+\mathbf{c}$,
2. $\underset{\mathbf{x}\to \mathbf{a}}{lim}\lambda \mathbf{f}(\mathbf{x})=\lambda \mathbf{b}$ for every $\lambda \in \mathbb{R}$,
3. $\underset{\mathbf{x}\to \mathbf{a}}{lim}\mathbf{f}(\mathbf{x})\cdot \mathbf{g}(\mathbf{x})=\mathbf{b}\cdot \mathbf{c}$,
4. $\underset{\mathbf{x}\to \mathbf{a}}{lim}\Vert \mathbf{f}(\mathbf{x})\Vert =\Vert \mathbf{b}\Vert$.

We prove a couple of the parts of the above theorem here, the other parts are left as exercises.

Proof of part 3.

Observe that $\mathbf{f}(\mathbf{x})\cdot \mathbf{g}(\mathbf{x})-\mathbf{b}\cdot \mathbf{c}=(\mathbf{f}(\mathbf{x})-\mathbf{b})\cdot (\mathbf{g}(\mathbf{x})-\mathbf{c})+\mathbf{b}\cdot (\mathbf{g}(\mathbf{x})-\mathbf{c})+\mathbf{c}\cdot (\mathbf{f}(\mathbf{x})-\mathbf{b})$. By the triangle inequality and Cauchy-Schwarz,

$$\begin{array}{rl}\Vert \mathbf{f}(\mathbf{x})\cdot \mathbf{g}(\mathbf{x})-\mathbf{b}\cdot \mathbf{c}\Vert & \le \Vert \mathbf{f}(\mathbf{x})-\mathbf{b}\Vert \Vert \mathbf{g}(\mathbf{x})-\mathbf{c}\Vert \\ & +\Vert \mathbf{b}\Vert \Vert \mathbf{g}(\mathbf{x})-\mathbf{c}\Vert \\ & +\Vert \mathbf{c}\Vert \Vert \mathbf{f}(\mathbf{x})-\mathbf{b}\Vert .\end{array}$$

Since we already know that $\Vert \mathbf{f}(\mathbf{x})-\mathbf{b}\Vert \to 0$ and $\Vert \mathbf{g}(\mathbf{x})-\mathbf{c}\Vert \to 0$ as $\mathbf{x}\to \mathbf{a}$, this implies that $\Vert \mathbf{f}(\mathbf{x})\cdot \mathbf{g}(\mathbf{x})-\mathbf{b}\cdot \mathbf{c}\Vert \to 0$.

Proof of part 4.

Take $\mathbf{f}=\mathbf{g}$ in part (c) implies that $\underset{\mathbf{x}\to \mathbf{a}}{lim}{\Vert \mathbf{f}(\mathbf{x})\Vert }^2={\Vert \mathbf{b}\Vert }^2$.

When writing a vector field (or similar functions) it is often convenient to divide the higher-dimensional function into smaller parts. We call these parts the components of a vector field. For example $\mathbf{F}(\mathbf{x})=(F_1(\mathbf{x}),F_2(\mathbf{x}))$ in 2D, $\mathbf{F}(\mathbf{x})=(F_1(\mathbf{x}),F_2(\mathbf{x}),F_3(\mathbf{x}))$ in 3D, etc.

Theorem

Let $\mathbf{F}(\mathbf{x})=(F_1(\mathbf{x}),F_2(\mathbf{x}))$. Then $\mathbf{F}$ is continuous if and only if $F_1$ and $F_2$ are continuous.

Proof

We will independently prove the two implications.

• ($\Rightarrow$) Let ${\mathbf{e}}_1=(1,0)$, ${\mathbf{e}}_2=(0,1)$ and observe that $F_k(\mathbf{x})=\mathbf{F}(\mathbf{x})\cdot {\mathbf{e}}_k$. We have already shown that the continuity of two vector fields implies the continuity of the inner product.
• ($\Leftarrow$) By definition of the norm$${\Vert \mathbf{F}(\mathbf{x})-\mathbf{F}(\mathbf{a})\Vert }^2=\sum _{k=1}^2{(F_k(\mathbf{x})-F_k(\mathbf{a}))}^2$$and we know $\Vert F_k(\mathbf{x})-F_k(\mathbf{a})\Vert \to 0$ as $\Vert \mathbf{x}-\mathbf{a}\Vert \to 0$.

In higher dimensions the analogous statement is true for the vector field $\mathbf{F}(\mathbf{x})=(F_1(\mathbf{x}),\dots ,F_m(\mathbf{x}))$ with exactly the same proof. I.e., $\mathbf{F}$ is continuous if and only if each $f_k$ is continuous.

Example (polynomials)

A polynomial in $n$ variables is a scalar field on ${\mathbb{R}}^n$ of the form

$$f(x_1,\dots ,x_n)=\sum _{k_1=0}^j\cdots \sum _{k_n=0}^j{c}_{k_1,\dots ,k_n}{x}_1^{k_1}\cdots x_n^{k_n}.$$

E.g., $f(x,y):=x+2xy-x^2$ is a polynomial in $2$ variables. Polynomials are continuous everywhere in ${\mathbb{R}}^n$. This is because they are the finite sum of products of continuous scalar fields.

Example (rational functions)

A rational function is a scalar field

$$f(\mathbf{x})=\frac{p(\mathbf{x})}{q(\mathbf{x})}$$

where $p(\mathbf{x})$ and $q(\mathbf{x})$ are polynomials. A rational function is continuous at every point $\mathbf{x}$ such that $q(\mathbf{x})\ne 0$.

As described in the following result, the continuity of functions continues to hold, in an intuitive way, under composition of functions.

Theorem

Suppose $S\subset {\mathbb{R}}^l$, $T\subset {\mathbb{R}}^m$, $\mathbf{f}:S\to {\mathbb{R}}^m$, $\mathbf{g}:T\to {\mathbb{R}}^n$ and that $\mathbf{f}(S)\subset T$ so that

$$(\mathbf{g}\circ \mathbf{f})(\mathbf{x})=\mathbf{g}(\mathbf{f}(\mathbf{x}))$$

makes sense. If $\mathbf{f}$ is continuous at $\mathbf{a}\in S$ and $\mathbf{g}$ is continuous at $\mathbf{f}(\mathbf{a})$ then $\mathbf{g}\circ \mathbf{f}$ is continuous at $\mathbf{a}$.

Proof

$$lim\mathbf{x}\to \mathbf{a}\Vert \mathbf{f}(\mathbf{g}(\mathbf{x}))-\mathbf{f}(\mathbf{g}(\mathbf{a}))\Vert =lim\mathbf{y}\to \mathbf{g}(\mathbf{a})\Vert \mathbf{f}(\mathbf{y})-\mathbf{f}(\mathbf{g}(\mathbf{a}))\Vert =0$$

Example

We can consider the scalar field $f(x,y)=\sin (x^2+y)+xy$ as the composition of functions.

Derivatives of scalar fields

Plot where colour represents the value of $f(x,y)=x^2+y^2$. The change in $f$ depends on direction

We can imagine, for example in the figure, that in higher dimensions, the derivative of a scalar field depends on the direction. This motivates the following.

Definition (directional derivative)

Let $S\subset {\mathbb{R}}^n$ and $f:S\to \mathbb{R}$. For any $\mathbf{a}\in \mathrm{int}S$ and $\mathbf{v}\in {\mathbb{R}}^n$, $\Vert v\Vert =1$ the directional derivative of $f$ with respect to $\mathbf{v}$ is defined as

$$D_{\mathbf{v}}f(\mathbf{a})=\underset{h\to 0}{lim}\frac{1}{h}(f(\mathbf{a}+h\mathbf{v})-f(\mathbf{a})).$$

When $h$ is small we can guarantee that $\mathbf{a}+h\mathbf{v}\in S$ because $\mathbf{a}\in \mathrm{int}S$ so this definition makes sense.

Theorem

Suppose $S\subset {\mathbb{R}}^n$, $f:S\to \mathbb{R}$, $\mathbf{a}\in \mathrm{int}S$. Let $g(t):=f(\mathbf{a}+t\mathbf{v})$. If one of the derivatives $g^{\prime }(t)$ or $D_{\mathbf{v}}f(\mathbf{a})$ exists then the other also exists and

$$g^{\prime }(t)=D_{\mathbf{v}}f(\mathbf{a}+t\mathbf{v}).$$

In particular $g^{\prime }(0)=D_{\mathbf{v}}f(\mathbf{a})$.

Proof

By definition $\frac{1}{h}(g(t+h)-g(h))=\frac{1}{h}(f(\mathbf{a}+h\mathbf{v})-f(\mathbf{a}))$.

The following result is useful for proving later results.

Theorem (mean value)

Assume that $D_{\mathbf{v}}(\mathbf{a}+t\mathbf{v})$ exists for each $t\in [0,1]$. Then for some $\theta \in (0,1)$,

$$f(\mathbf{a}+\mathbf{v})-f(\mathbf{a})=D_{\mathbf{v}}f(\mathbf{z}),\text{where }z=\mathbf{a}+\theta \mathbf{v}.$$

Proof

Apply mean value theorem to $g(t)=f(\mathbf{a}+t\mathbf{v})$.

The following notation is convenient. For any $k\in \{1,2,\dots ,n\}$, let ${\mathbf{e}}_k$ be the $n$-dimensional unit vector where all entries are zero except the $k$th position which is equal to $1$. I.e., ${\mathbf{e}}_1=(1,0,\dots ,0)$, ${\mathbf{e}}_2=(0,1,0,\dots ,0)$, ${\mathbf{e}}_n=(0,\dots ,0,1)$.

Definition (partial derivatives)

We define the partial derivative in $x_k$ of $f(x_1,\dots ,x_n)$ at $\mathbf{a}$ as

$$\frac{\partial f}{\partial x_k}(\mathbf{a})=D_{{\mathbf{e}}_k}f(\mathbf{a}).$$

Remark

Various symbols used for partial derivatives: $\frac{\partial f}{\partial x_k}(\mathbf{a})=D_{k}f(\mathbf{a})={\partial }_{k}f(\mathbf{a})$. If a function is written $f(x,y)$ we write $\frac{\partial f}{\partial x},\frac{\partial f}{\partial y}$ for the partial derivatives. Similarly for higher dimension.

In practice, to compute the partial derivative $\frac{\partial f}{\partial x_k}$, one should consider all other $x_j$ for $j\ne k$ as constants and take the derivative with respect to $x_k$. In a moment we see this rigorously.

If $f:\mathbb{R}\to \mathbb{R}$ is differentiable, then we know that, when $x$ is close to $a$,

$$f(x)\approx f(a)+(x-a)f^{\prime }(a).$$

More precisely, we know that $f(x)=f(a)+(x-a)f^{\prime }(a)+ϵ(x-a)$ where $|ϵ(x-a)|=o(|x-a|)$. (This is little-o notation and here means that $|f(x)-f(a)-(x-a)f^{\prime }(a)|/|x-a|\to 0$ as $|x-a|\to 0$.) This way of seeing differentiability is convenient for the higher dimensional definition of differentiability.

Definition (differentiable)

Let $S\subset {\mathbb{R}}^n$ be open, $f:S\to \mathbb{R}$. We say that $f$ is differentiable at $\mathbf{a}\in S$ if there exists a linear transformation ${df}_{\mathbf{a}}:{\mathbb{R}}^n\to \mathbb{R}$ such that, for $\mathbf{x}\in B(\mathbf{a},r)$,

$$f(\mathbf{x})=f(\mathbf{a})+{df}_{\mathbf{a}}(\mathbf{x}-\mathbf{a})+ϵ(\mathbf{x}-\mathbf{a})$$

where $|ϵ(\mathbf{x}-\mathbf{a})|=o(\Vert \mathbf{x}-\mathbf{a}\Vert )$.

For future convenience we introduce the following notation.

Definition (gradient)

The gradient of the scalar field $f(x,y,z)$ at the point $\mathbf{a}$ is

$$\mathrm{\nabla }f(\mathbf{a})=\left(\begin{array}{c}\frac{\partial f}{\partial x}(\mathbf{a})\\ \frac{\partial f}{\partial y}(\mathbf{a})\\ \frac{\partial f}{\partial z}(\mathbf{a})\end{array}\right).$$

In general, when working in ${\mathbb{R}}^n$ for some $n\in \mathbb{N}$, the gradient of the scalar field $f(x_1,\dots ,x_n)$ at the point $\mathbf{a}$ is

$$\mathrm{\nabla }f(\mathbf{a})=\left(\begin{array}{c}\frac{\partial f}{\partial x_1}(\mathbf{a})\\ \frac{\partial f}{\partial x_2}(\mathbf{a})\\ ⋮\\ \frac{\partial f}{\partial x_n}(\mathbf{a})\end{array}\right).$$

Theorem

If $f$ is differentiable at $\mathbf{a}$ then ${df}_{\mathbf{a}}(\mathbf{v})=\mathrm{\nabla }f(\mathbf{a})\cdot \mathbf{v}$. This means that, for $\mathbf{x}\in B(\mathbf{a},r)$,

$$f(\mathbf{x})=f(\mathbf{a})+\mathrm{\nabla }f(\mathbf{a})\cdot (\mathbf{x}-\mathbf{a})+ϵ(\mathbf{x}-\mathbf{a})$$

where $|ϵ(\mathbf{x}-\mathbf{a})|=o(\Vert \mathbf{x}-\mathbf{a}\Vert )$. Moreover, for any vector $\mathbf{v}$, $\Vert v\Vert =1$,

$$D_{\mathbf{v}}f(\mathbf{a})=\mathrm{\nabla }f(\mathbf{a})\cdot \mathbf{v}.$$

Proof

Since $f$ is differentiable there exists a linear transformation ${df}_{\mathbf{a}}:{\mathbb{R}}^n\to \mathbb{R}$ such that $f(\mathbf{a}+h\mathbf{v})=f(\mathbf{a})+h{df}_{\mathbf{a}}(\mathbf{v})+ϵ(h\mathbf{v})$ and hence

$$\begin{array}{rl}{D}_{\mathbf{v}}f(\mathbf{a})& =\underset{h\to 0}{lim}\frac{1}{h}(f(\mathbf{a}+h\mathbf{v})-f(\mathbf{a}))\\ & =\underset{h\to 0}{lim}\frac{1}{h}(h{df}_{\mathbf{a}}(\mathbf{v})+ϵ(h\mathbf{v}))={df}_{\mathbf{a}}(\mathbf{v}).\end{array}$$

In particular ${df}_{\mathbf{a}}({\mathbf{e}}_k)=D_{{\mathbf{e}}_k}f(\mathbf{a})$.

Theorem

If $f$ is differentiable at $\mathbf{a}$, then it is continuous at $\mathbf{a}$.

Proof

Observe that $|f(\mathbf{a}+\mathbf{v})-f(\mathbf{a})|=|{df}_{\mathbf{a}}(\mathbf{v})+ϵ(\mathbf{v})|$. This means that

$$|f(\mathbf{a}+\mathbf{v})-f(\mathbf{a})|\le \Vert {df}_{\mathbf{a}}\Vert \Vert \mathbf{v}\Vert +|ϵ(\mathbf{v})|$$

and so this tends to $0$ as $\Vert \mathbf{v}\Vert \to 0$.

Theorem

Suppose that $f(x_1,\dots ,x_n)$ is a scalar field. If the partial derivatives ${\partial }_{1}f(\mathbf{x}),\dots ,{\partial }_{n}f(\mathbf{x})$ exist for all $\mathbf{x}\in B(\mathbf{a},r)$ and are continuous at $\mathbf{a}$ then $f$ is differentiable at $\mathbf{a}$.

Proof

For convenience define the vectors

$$\begin{array}{rl}\mathbf{v}& =(v_1,v_2,\dots ,v_n),\\ {\mathbf{u}}_k& =(v_1,v_2,\dots ,v_k,0,\dots ,0).\end{array}$$

Observe that

$${\mathbf{u}}_k-{\mathbf{u}}_{k-1}=v_k{\mathbf{e}}_k,{\mathbf{u}}_0=(0,0,\dots ,0),{\mathbf{u}}_n=\mathbf{v}.$$

Using the mean value theorem we know that there exists ${\mathbf{z}}_k={\mathbf{u}}_{k-1}+{\theta }_k{\mathbf{e}}_k$ such that $f(\mathbf{a}+{\mathbf{u}}_k)-f(\mathbf{a}+{\mathbf{u}}_{k-1})=v_k{D}_{\mathbf{e}\mathrm{\_}k}f(\mathbf{a}+\mathbf{z}\mathrm{\_}k)$. Consequently

$$\begin{array}{rl}f(\mathbf{a}+\mathbf{v})-f(\mathbf{a})& =\sum _{k=1}^{n}f(\mathbf{a}+{\mathbf{u}}_k)-f(\mathbf{a}+{\mathbf{u}}_{k-1})\\ & =\sum _{k=1}^n{v}_k{D}_{{\mathbf{e}}_k}f(\mathbf{a}+\mathbf{z}\mathrm{\_}k)\\ & =\sum _{k=1}^n{v}_k{D}_{{\mathbf{e}}_k}f(\mathbf{a}+{\mathbf{u}}_{k-1})\\ & +\sum _{k=1}^n{v}_k(D_{{\mathbf{e}}_k}f(\mathbf{a}+\mathbf{z}\mathrm{\_}k)-D_{{\mathbf{e}}_k}f(\mathbf{a}+{\mathbf{u}}_{k-1}))\end{array}$$

To conclude, observe that the second sum vanishes as $\Vert \mathbf{v}\Vert \to 0$ and that the first sum, $\sum _{k=1}^n{v}_k{D}_{{\mathbf{e}}_k}f(\mathbf{a}+{\mathbf{u}}_{k-1})$, converges to $\mathbf{v}\cdot \mathrm{\nabla }f(\mathbf{a})$.

Chain rule

When we are working in $\mathbb{R}$ we know that, if $g$ and $h$ are differentiable, then $f(t)=g\circ h(t)$ is also differentiable and also $f^{\prime }(t)=g^{\prime }(h(t))h^{\prime }(t)$. This is called the chain rule and is frequently very useful in calculating derivatives. We now investigate how this extends to higher dimension?

Example

Suppose that $\mathit{\alpha }:\mathbb{R}\to {\mathbb{R}}^3$ describes the position $\mathit{\alpha }(t)$ at time $t$ and that $f:{\mathbb{R}}^3\to \mathbb{R}$ describes the temperature $f(\mathit{\alpha })$ at a point $\mathit{\alpha }$ The temperature at time $t$ is equal to $g(t)=f(\mathit{\alpha }(t))$. We want to calculate $g^{\prime }(t)$ because this is the change in temperature with respect to time.

In situations like the above example it is convenient to consider the derivative of a path $\mathit{\alpha }:\mathbb{R}\to {\mathbb{R}}^n$. Let $\mathit{\alpha }:\mathbb{R}\to {\mathbb{R}}^n$ and suppose it has the form $\mathit{\alpha }(t)=({\alpha }_1(t),\dots ,{\alpha }_n(t))$. We define the derivative as

$${\mathit{\alpha }}^{\prime }(t):=\left(\begin{array}{c}{x}_1^{\prime }(t)\\ ⋮\\ x_n^{\prime }(t)\end{array}\right).$$

Here ${\mathit{\alpha }}^{\prime }$ is a vector-valued function which represents the "direction of movement".

$\mathit{\alpha }(t)=(\cos t,\sin t,t)$, $t\in \mathbb{R}$

Theorem

Let $S\subset {\mathbb{R}}^n$ be open and $I\subset \mathbb{R}$ an interval. Let $\mathbf{x}:I\to S$ and $f:S\to \mathbb{R}$ and define, for $t\in I$,

$$g(t)=f(\mathbf{x}(t)).$$

Suppose that $t\in I$ is such that ${\mathbf{x}}^{\prime }(t)$ exists and $f$ is differentiable at $\mathbf{x}(t)$. Then $g^{\prime }(t)$ exists and

$$g^{\prime }(t)=\mathrm{\nabla }f(\mathbf{x}(t))\cdot {\mathbf{x}}^{\prime }(t).$$

Proof

Since $f$ is differentiable, $f(\mathbf{y})-f(\mathbf{x})=\mathrm{\nabla }f(\mathbf{x})\cdot (\mathbf{y}-\mathbf{x})+ϵ(\mathbf{x},\mathbf{y}-\mathbf{x})$ where $|ϵ(\mathbf{x},\mathbf{y}-\mathbf{x})|=o(\mathbf{y}-\mathbf{x})$. Let $h>0$ be small.

$$\begin{array}{rl}\frac{1}{h}[g(t+h)-g(t)]& =\frac{1}{h}[f(\mathbf{x}(t+h))-f(\mathbf{x}(t))]\\ & =\frac{1}{h}\mathrm{\nabla }f(\mathbf{x}(t))\cdot (\mathbf{x}(t+h)-\mathbf{x}(t))\\ & +\frac{1}{h}ϵ(\mathbf{x}(t),\mathbf{x}(t+h)-\mathbf{x}(t)).\end{array}$$

Observe that $\frac{1}{h}(\mathbf{x}(t+h)-\mathbf{x}(t))\to {\mathbf{x}}^{\prime }(t)$ as $h\to 0$.

Example

A particle moves in a circle and its position at time $t\in [0,2\pi ]$ is given by

$$\mathbf{x}(t)=(\cos t,\sin t).$$

The temperature at a point $\mathbf{y}=(y_1,y_2)$ is given by the function $f(\mathbf{y}):=y_1+y_2$, The temperature the particle experiences at time $t$ is given by $g(t)=f(\mathbf{x}(t))$. Temperature change:

$$g^{\prime }(t)=\mathrm{\nabla }f(\mathbf{x}(t))\cdot {\mathbf{x}}^{\prime }(t)=\left(\begin{array}{c}1\\ 1\end{array}\right)\cdot \left(\begin{array}{c}-\sin t\\ \cos t\end{array}\right)=\cos t-\sin t.$$

$\mathbf{x}(t)$ is the position of a particle.

Level sets & tangent planes

Let $S\subset {\mathbb{R}}^2$, $f:S\to \mathbb{R}$. Suppose $c\in \mathbb{R}$ and let

$$L(c)=\{\mathbf{x}\in S:f(\mathbf{x})=c\}.$$

The set $L(c)$ is called the level set. In general this set can be empty or it can be all of $S$. However the set $L(c)$ is often a curve and this is the case of interest. This is the same notion as that of contour lines on a map. I.e., $\mathbf{x}(t_a)=\mathbf{a}$ for some $t_a\in I$ and

$$f(\mathbf{x}(t))=c$$

for all $t\in I$. Then

• $\mathrm{\nabla }f(\mathbf{a})$ is normal to the curve at $\mathbf{a}$
• Tangent line at $\mathbf{a}$ is $\{\mathbf{x}\in {\mathbb{R}}^2:\mathrm{\nabla }f(\mathbf{a})\cdot (\mathbf{x}-\mathbf{a})=0\}$

This is because the chain rule implies that $\mathrm{\nabla }f(\mathbf{x}(t))\cdot {\mathbf{x}}^{\prime }(t)=0$.

Example

Let $f(x_1,x_2,x_3):=x_1^2+x_2^2+x_3^2$.

• If $c>0$ then $L(c)$ is a sphere,
• $L(0)$ is a single point $(0,0,0)$,
• If $c<0$ then $L(c)$ is empty.

Example

Let $f(x_1,x_2,x_3):=x_1^2+x_2^2-x_3^2$. See figure.

• If $c>0$ then $L(c)$ is a one-sheeted hyperboloid,
• $L(0)$ is an infinite cone,
• If $c<0$ then $L(c)$ is a two-sheeted hyperboloid.

Sphere

2-sheet hyperboloid

Infinite cone

1-sheet hyperboloid

Let $f$ be a differentiable scalar field on $S\subset {\mathbb{R}}^3$ and suppose that the level set $L(c)=\{\mathbf{x}\in S:f(\mathbf{x})=c\}$ defines a surface.

• The gradient $\mathrm{\nabla }f(\mathbf{a})$ is normal to every curve $\mathit{\alpha }(t)$ in the surface which passes through $\mathbf{a}$,
• The tangent plane at $\mathbf{a}$ is $\{\mathbf{x}\in {\mathbb{R}}^3:\mathrm{\nabla }f(\mathbf{a})\cdot (\mathbf{x}-\mathbf{a})=0\}$.

Same argument as in ${\mathbb{R}}^2$ works in ${\mathbb{R}}^n$.

Tangent plane and normal vector

Derivatives of vector fields

Essentially everything discussed above for scalar fields extends to vector fields in a predictable way. This is because of the linearity and that we can consider each component of the vector field independently.

Definition (directional derivative)

Let $S\subset {\mathbb{R}}^n$ and $\mathbf{F}:S\to {\mathbb{R}}^m$. For any $\mathbf{a}\in \mathrm{int}S$ and $\mathbf{v}\in {\mathbb{R}}^n$ the derivative of the vector field $\mathbf{F}$ with respect to $\mathbf{v}$ is defined as

$$D_{\mathbf{v}}\mathbf{F}(\mathbf{a}):=\underset{h\to 0}{lim}\frac{1}{h}(\mathbf{F}(\mathbf{a}+h\mathbf{v})-\mathbf{F}(\mathbf{a})).$$

Remark

If we use the notation $\mathbf{F}=(F_1,\dots ,F_m)$, i.e., we write the function using the ``components'' where each $F_k$ is a scalar field, then $D_{\mathbf{v}}\mathbf{F}=(D_{\mathbf{v}}{F}_1,\dots ,D_{\mathbf{v}}{F}_m)$.

Definition (differentiable)

We say that $\mathbf{F}:{\mathbb{R}}^n\to {\mathbb{R}}^m$ is differentiable at $\mathbf{a}$ if there exists a linear transformation ${df}_{\mathbf{a}}:{\mathbb{R}}^n\to {\mathbb{R}}^m$ such that, for $\mathbf{x}\in B(\mathbf{a},r)$,

$$\mathbf{F}(\mathbf{x})=\mathbf{F}(\mathbf{a})+{df}_{\mathbf{a}}(\mathbf{x}-\mathbf{a})+ϵ(\mathbf{x}-\mathbf{a}),$$

$\Vert ϵ(\mathbf{x}-\mathbf{a})\Vert =o(\Vert \mathbf{x}-\mathbf{a}\Vert )$.

Theorem

If $\mathbf{F}$ is differentiable at $\mathbf{a}$ then $\mathbf{F}$ is continuous at $\mathbf{a}$ and ${df}_{\mathbf{a}}(\mathbf{v})=D_{\mathbf{v}}\mathbf{F}(\mathbf{a})$.

Proof

Same as for the case of scalar fields when $f:{\mathbb{R}}^n\to \mathbb{R}$.

Jacobian matrix & the chain rule

The relevant differential for higher-dimensional functions is the Jacobian matrix.

Definition (Jacobian matrix)

Suppose that $\mathbf{F}:{\mathbb{R}}^2\to {\mathbb{R}}^2$ and use the notation $\mathbf{F}(x,y)=(F_1(x,y),F_2(x,y))$. The Jacobian matrix of $\mathbf{F}$ at $\mathbf{a}$ is defined as

$$D\mathbf{F}(\mathbf{a})=\left(\begin{array}{cc}\frac{\partial F_1}{\partial x}(\mathbf{a})& \frac{\partial F_1}{\partial y}(\mathbf{a})\\ \frac{\partial F_2}{\partial x}(\mathbf{a})& \frac{\partial F_2}{\partial y}(\mathbf{a})\end{array}\right).$$

The Jacobian matrix is defined analogously in any dimension. I.e., if $\mathbf{F}:{\mathbb{R}}^n\to {\mathbb{R}}^m$ the the Jacobian at $\mathbf{a}$ is

$$D\mathbf{F}(\mathbf{a})=\left(\begin{array}{cccc}{\partial }_1{F}_1(\mathbf{a})& {\partial }_2{F}_1(\mathbf{a})& \cdots & {\partial }_n{F}_1(\mathbf{a})\\ {\partial }_1{F}_2(\mathbf{a})& {\partial }_2{F}_2(\mathbf{a})& \cdots & {\partial }_n{F}_2(\mathbf{a})\\ ⋮& ⋮& & ⋮\\ {\partial }_1{F}_m(\mathbf{a})& {\partial }_2{F}_m(\mathbf{a})& \cdots & {\partial }_n{F}_m(\mathbf{a})\end{array}\right)$$

If we choose a basis then any linear transformation ${\mathbb{R}}^n\to {\mathbb{R}}^m$ can be written as a $m\times n$ matrix. We find that ${df}_{\mathbf{a}}(\mathbf{v})=D\mathbf{F}(\mathbf{a})\mathbf{v}$.

Let $S\subset {\mathbb{R}}^n$ and $\mathbf{F}:S\to {\mathbb{R}}^m$. If $f$ is differentiable at $\mathbf{a}\in S$ then, for all $\mathbf{x}\in B(\mathbf{a},r)\subset S$,

$$\mathbf{F}(\mathbf{x})=\mathbf{F}(\mathbf{a})+D\mathbf{F}(\mathbf{a})(\mathbf{x}-\mathbf{a})+ϵ(\mathbf{x}-\mathbf{a})$$

where $|ϵ(\mathbf{x}-\mathbf{a})|=o(\Vert \mathbf{x}-\mathbf{a}\Vert )$. This is like a Taylor expansion in higher dimensions.

Here we see that in higher dimensions we have a matrix form of the chain rule.

Theorem

Let $S\subset {\mathbb{R}}^l$, $T\subset {\mathbb{R}}^m$ be open. Let $\mathbf{f}:S\to T$ and $\mathbf{g}:T\to {\mathbb{R}}^n$ and define

$$\mathbf{h}=\mathbf{g}\circ \mathbf{f}:S\to {\mathbb{R}}^n.$$

Let $\mathbf{a}\in S$. Suppose that $\mathbf{f}$ is differentiable at $\mathbf{a}$ and $\mathbf{g}$ is differentiable at $\mathbf{f}(\mathbf{a})$. Then $\mathbf{h}$ is differentiable at $\mathbf{a}$ and

$$D\mathbf{h}(\mathbf{a})=D\mathbf{g}(\mathbf{f}(\mathbf{a}))D\mathbf{f}(\mathbf{a}).$$

Proof

Let $\mathbf{u}=\mathbf{f}(\mathbf{a}+\mathbf{v})-\mathbf{f}(\mathbf{a})$. Since $\mathbf{f}$ and $\mathbf{g}$ are differentiable,

$$\begin{array}{rl}\mathbf{h}(\mathbf{a}+\mathbf{v})-\mathbf{h}(\mathbf{a})& =\mathbf{g}(\mathbf{f}(\mathbf{a}+\mathbf{v}))-\mathbf{g}(\mathbf{f}(\mathbf{a}))\\ & =D\mathbf{g}(\mathbf{f}(\mathbf{a}))(\mathbf{f}(\mathbf{a}+\mathbf{v})-\mathbf{f}(\mathbf{a}))+{\mathit{ϵ}}_{\mathbf{g}}(\mathbf{u})\\ & =D\mathbf{g}(\mathbf{f}(\mathbf{a}))D\mathbf{f}(\mathbf{a})\mathbf{v}+D\mathbf{g}(\mathbf{f}(\mathbf{a})){\mathit{ϵ}}_{\mathbf{f}}(\mathbf{v})+{\mathit{ϵ}}_{\mathbf{g}}(\mathbf{u}).\end{array}$$

Example (polar coordinates)

Here we consider polar coordinates and calculate the Jacobian of this transformation. We can write the change of coordinates

$$(r,\theta )\mapsto (r\cos \theta ,r\sin \theta )$$

as the function $\mathbf{f}(r,\theta )=(x(r,\theta ),y(r,\theta ))$ where $\mathbf{f}:(0,\mathrm{\infty })\times [0,2\pi )\to {\mathbb{R}}^2$. We calculate the Jacobian matrix of this transformation

$$D\mathbf{f}(r,\theta )=\left(\begin{array}{cc}\frac{\partial x}{\partial r}(r,\theta )& \frac{\partial x}{\partial \theta }(r,\theta )\\ \frac{\partial y}{\partial r}(r,\theta )& \frac{\partial y}{\partial \theta }(r,\theta )\end{array}\right)=\left(\begin{array}{cc}\cos \theta & -r\sin \theta \\ \sin \theta & r\cos \theta \end{array}\right).$$

In particular we see that $detD\mathbf{f}(r,\theta )=r$, the familiar value used in change of variables with polar coordinated.

Suppose now that we wish to calculate derivatives of $h:=g\circ \mathbf{f}$ for some $g:{\mathbb{R}}^2\to \mathbb{R}$. Here we take advantage of the theorem concerning multiplication of Jacobians.

$$\begin{array}{rl}Dh(r,\theta )& =Dg(\mathbf{f}(r,\theta ))D\mathbf{f}(r,\theta )\\ \left(\begin{array}{cc}\frac{\partial h}{\partial r}(r,\theta )& \frac{\partial h}{\partial \theta }(r,\theta )\end{array}\right)& =\left(\begin{array}{cc}\frac{\partial g}{\partial x}(\mathbf{f}(r,\theta ))& \frac{\partial g}{\partial y}(\mathbf{f}(r,\theta ))\end{array}\right)\left(\begin{array}{cc}\cos \theta & -r\sin \theta \\ \sin \theta & r\cos \theta \end{array}\right)\end{array}$$

In other words, we have shown that

$$\begin{array}{rl}\frac{\partial h}{\partial r}(r,\theta )& =\frac{\partial g}{\partial x}(r\cos \theta ,r\sin \theta )\cos \theta +\frac{\partial g}{\partial y}(r\cos \theta ,r\sin \theta )\sin \theta \\ \frac{\partial h}{\partial \theta }(r,\theta )& =-r\frac{\partial g}{\partial x}(r\cos \theta ,r\sin \theta )\sin \theta +r\frac{\partial g}{\partial y}(r\cos \theta ,r\sin \theta )\cos \theta .\end{array}$$

Implicit functions & partial derivatives

Just like with derivatives, we can take higher order partial derivatives. For convenience when we want to write $\frac{\partial }{\partial y}\frac{\partial }{\partial x}f(x,y)$, i.e., differentiate first with respect to $x$ and then with respect to $y$, we write instead $\frac{{\partial }^{2}f}{\partial y\partial x}(x,y)$. The analogous notation is used for higher derivatives and any other choice of coordinates.

We first consider the question of when

$$\frac{{\partial }^{2}f}{\partial y\partial x}(x,y)\stackrel{\text{?}}{=}\frac{{\partial }^{2}f}{\partial x\partial y}(x,y).$$

Example (partial derivative problem)

Let $f:{\mathbb{R}}^2\to \mathbb{R}$ be defined as $f(0,0)=0$ and, for $(x,y)\ne (0,0)$,

$$f(x,y):=\frac{xy(x^2-y^2)}{x^2+y^2}.$$

We calculate that $\frac{{\partial }^{2}f}{\partial y\partial x}(0,0)=-1$ but $\frac{{\partial }^{2}f}{\partial x\partial y}(0,0)=1$.

Theorem

Let $f:S\to \mathbb{R}$ be a scalar field such that the partial derivatives $\frac{\partial f}{\partial x}$, $\frac{\partial f}{\partial y}$ and $\frac{{\partial }^{2}f}{\partial y\partial x}$ exist on an open set $S\subset {\mathbb{R}}^2$ containing $\mathbf{x}$. Further assume that $\frac{{\partial }^{2}f}{\partial y\partial x}$ is continuous on $S$. Then the derivative $\frac{{\partial }^{2}f}{\partial x\partial y}(\mathbf{x})$ exists and

$$\frac{{\partial }^{2}f}{\partial x\partial y}(\mathbf{x})=\frac{{\partial }^{2}f}{\partial y\partial x}(\mathbf{x}).$$

In many cases we can choose to write a given curve/function either in implicit or explicit form.

Implicit	Explicit
$x^2-y=0$	$y(x)=x^2$
$x^2+y^2=1$	$y(x)=\pm \sqrt{1-x^2}$, $\left|x\right|\le 1$
$x^2-y^2-1=0$	$y(x)=\pm \sqrt{x^2-1}$, $\left|x\right|\ge 1$
$x^2+y^2-e^y-4=0$	A mess?
$x^2{y}^4-3=\sin (xy)$	A huge mess?

Given the above observation, the following method of calculating derivatives is sometimes useful. Suppose that some $f:{\mathbb{R}}^2\to \mathbb{R}$ is given and we suppose there exists some $y:\mathbb{R}\to \mathbb{R}$ such that

$$f(x,y(x))=0\text{ for all }x.$$

Let $h(x):=f(x,y(x))$ and note that $h^{\prime }(x)=0$. Here we are using the idea that $h=f\circ g$ where $g(x)=(x,y(x))$. By the chain rule $h^{\prime }(x)$ is equal to

$$\left(\begin{array}{cc}\frac{\partial f}{\partial x}(x,y(x))& \frac{\partial f}{\partial y}(x,y(x))\end{array}\right)\left(\begin{array}{c}1\\ y^{\prime }(x)\end{array}\right)=0.$$

Consequently

$$y^{\prime }(x)=-\frac{\frac{\partial f}{\partial x}(x,y(x))}{\frac{\partial f}{\partial y}(x,y(x))}.$$