Calculus

All things calculus, differential or integral.

Definitions & Formulas

Differentiation

As the name states, we analyze differences in the output(s) of a function based on differences in the input(s) \[ \dfrac{dy}{dx} = \lim_{h \rightarrow 0} \dfrac{f(x + h) - f(x)}{h} \]

A function is said to be differentiable at \(x\) if this limit exists, and differentiable on an interval if it exists at any \(x\) in this interval.

Common Derivatives

From that we can get common derivatives

\[ \begin{aligned} \dfrac{d}{dx}C & = 0 && \text{for any constant C} \\ \dfrac{d}{dx}x^n & = nx^{n - 1} && \text{for n} \neq 0 \\ \dfrac{d}{dx}e^x & = e^x \\ \dfrac{d}{dx}\ln x & = x^{-1} \\ \dfrac{d}{dx}a^x & = \ln(a)a^x \\ \dfrac{d}{dx}\cos x & = -\sin x \\ \dfrac{d}{dx}\sin x & = \cos x \\ \end{aligned} \]

From these it becomes trivial to derive \(\tan\), \(\sec\), \(\csc\) and \(\cot\).

Derivation Rules

Finally a composition of differentiable functions is also differentiable, so we have the following rules that allow us to derive almost any function:

\[ \begin{aligned} \dfrac{d}{dx}Cf(x) & = C\dfrac{d}{dx}f(x) && \text{Constant multiple rule} \\ \dfrac{d}{dx}[f(x) + g(x)] & = \dfrac{d}{dx}f(x) + \dfrac{d}{dx}g(x) && \text{Sum rule} \\ \dfrac{d}{dx}[f(x)g(x)] & = \dfrac{d}{dx}f(x)g(x) + f(x)\dfrac{d}{dx}g(x) && \text{Product rule} \\ \dfrac{dy}{dx} & = \dfrac{dy}{dz}\dfrac{dz}{dx} = \dfrac{\frac{dy}{dz}}{\frac{dx}{dz}} && \text{Chain rule} \\ \end{aligned} \]

From these we can easily find a Quotient Rule, a Power Rule and a Reciprocal Rule:

\[ \begin{aligned} \dfrac{d}{dx}\dfrac{f(x)}{g(x)} & = \dfrac{\frac{d}{dx}f(x)g(x) - f(x)\frac{d}{dx}g(x)}{g(x)^2} && \text{Quotient rule} \\ \dfrac{d}{dx}f(x)^n & = nf(x)^{n-1}\dfrac{d}{dx}f(x) && \text{Power rule} \\ \dfrac{dy}{dx}\dfrac{1}{f(x)} & = -\dfrac{\frac{d}{dx}f(x)}{f(x)^2} && \text{Reciprocal rule} \\ \end{aligned} \]

Note

Because of the definition of derivative as a rate of change, this is possible \(\dfrac{dy}{dx} = \dfrac{1}{\frac{dx}{dy}}\)

Multivariate Calculus

Gradient

Very similar to univariate calculus, but now our function takes a vector \(\mathbf{x} \in \mathbb{R}^n\) as input and returns a scalar \(y \in \mathbb{R}\).

To paraphrase D2L because their explanation is perfect:

Let \(y = f(x_1, x_2, \ldots, x_n)\) be a function with \(n\) variables. The partial derivative of \(y\) with respect to its \(i^\textrm{th}\) parameter \(x_i\) is

\[ \dfrac{\partial y}{\partial x_i} = \lim_{h \rightarrow 0} \frac{f(x_1, \ldots, x_{i-1}, x_i+h, x_{i+1}, \ldots, x_n) - f(x_1, \ldots, x_i, \ldots, x_n)}{h}.\]

For \(f \colon \mathbb{R}^n \to \mathbb{R}\), we collect/concatenate all the partial derivatives to obtain the gradient of the output \(y = f(\mathbf{x})\) with respect to the input \(\mathbf{x}\) \[ \nabla_{\mathbf{x}}f(\mathbf{x}) = \nabla_{\mathbf{x}}y =\begin{bmatrix} \frac{\partial y}{\partial x_1} \\ \vdots \\ \frac{\partial y}{\partial x_n} \end{bmatrix} \] sometimes written \(\nabla f(\mathbf{x})\) or \(\nabla y\) when not ambiguous.

Jacobian

The Jacobian \(\mathbf{J} \in \mathbb{R}^{m \times n}\) is a generalization of the gradient to \(\mathbf{y} = f(\mathbf{x})\) with \(f \colon \mathbb{R}^n \to \mathbb{R}^m\), where \(j_{i,j} = \dfrac{\partial y_i}{\partial\mathbf{x}_j}\), \(\mathbf{x} = \begin{bmatrix} x_1 \\ \vdots \\ x_n \end{bmatrix} \in \mathbb{R}^n\) and \(\mathbf{y} = \begin{bmatrix} y_1 \\ \vdots \\ y_m \end{bmatrix} \in \mathbb{R}^m\).

Explicitly \[ \displaystyle{ \mathbf{J} = \begin{bmatrix} \dfrac{\partial \mathbf{y}}{\partial x_{1}} & \cdots & \dfrac{\partial \mathbf{y}}{\partial x_{n}} \end{bmatrix} = \begin{bmatrix} \nabla^{\top}y_{1} \\ \vdots \\ \nabla^{\top}y_{m} \end{bmatrix} = \begin{bmatrix} \dfrac{\partial y_{1}}{\partial x_{1}} & \cdots & \dfrac{\partial y_{1}}{\partial x_{n}} \\ \vdots & \ddots & \vdots \\ \dfrac{\partial y_{m}}{\partial x_{1}} & \cdots & \dfrac{\partial y_{m}}{\partial x_{n}} \end{bmatrix}} \]

Handy Rules

The following rules come straight from D2L:

• For all \(\mathbf{A} \in \mathbb{R}^{m \times n}\) we have \(\nabla_{\mathbf{x}} \mathbf{A} \mathbf{x} = \mathbf{A}^\top\) and \(\nabla_{\mathbf{x}} \mathbf{x}^\top \mathbf{A} = \mathbf{A}\).
• For square matrices \(\mathbf{A} \in \mathbb{R}^{n \times n}\) we have that \(\nabla_{\mathbf{x}} \mathbf{x}^\top \mathbf{A} \mathbf{x} = (\mathbf{A} + \mathbf{A}^\top)\mathbf{x}\) and in particular \(\nabla_{\mathbf{x}} \|\mathbf{x} \|^2 = \nabla_{\mathbf{x}} \mathbf{x}^\top \mathbf{x} = 2\mathbf{x}\).

Then the chain rule states that

\[\frac{\partial y}{\partial x_{i}} = \frac{\partial y}{\partial u_{1}} \frac{\partial u_{1}}{\partial x_{i}} + \frac{\partial y}{\partial u_{2}} \frac{\partial u_{2}}{\partial x_{i}} + \ldots + \frac{\partial y}{\partial u_{m}} \frac{\partial u_{m}}{\partial x_{i}} \ \textrm{ and so } \ \nabla_{\mathbf{x}} y = \mathbf{A} \nabla_{\mathbf{u}} y,\]

where \(\mathbf{A} \in \mathbb{R}^{n \times m}\) is a matrix that contains the derivative of vector \(\mathbf{u}\) with respect to vector \(\mathbf{x}\).

Integrals

Integrals are

• a way to compute the signed area under a curve
• antiderivatives
• a way of adding up tiny bits

\[ \int_a^b f(x)\,dx \]

Intuition as sum of rectangles

The integral \[\int_a^b f(x)\,dx\] is the limit of sums of tiny rectangular areas.

If we cut the interval \([a,b]\) into \(n\) equal chunks of width \[\Delta x = \frac{b-a}{n},\] then the total area is approximated by \[\sum_{k=1}^{n} f(a + k\Delta x)\,\Delta x.\]

As we make the chunks thinner (\(n \to \infty\), so \(\Delta x \to 0\)), this sum becomes exact:

\[\int_a^b f(x)\,dx = \lim_{n\to\infty} \sum_{k=1}^{n} f(a + k\Delta x)\,\Delta x.\]

\(\int_a^b f(x)\,dx\) is a definite integral of \(f(x)\) from \([a,b]\).

\(\int f(x)\,dx\) is an indefinite integral.

Fundamental Theorem of Calculus

The fundamental theorem of calculus links differentiation (derivatives) and integration (integrals).

\[ \int_a^b f(x)\,dx = F(b) - F(a)\]

Where

\[ \dfrac{d}{dx}F(x) = f(x) \]

Intuition

Say we have a function \(A(x)\) being the area under the curve of \(f(x)\) between \(0\) and \(x\).

to find the area under the curve between \(x\) and \(x+h\), we could compute \[ A(x+h) - A(x) \approx f(x)h \] \[ \iff \dfrac{A(x+h) - A(x)}{h} \approx f(x) \] \[ \iff \lim_{h \to 0} \dfrac{A(x+h) - A(x)}{h} = f(x) \] \[ \iff \dfrac{d}{dx}A(x) = f(x) \]

dx

\(dx\) = the differential of \(x\)

A single symbol that means “infinitesimal change in \(x\).”

In derivatives, it appears in a ratio (\(\dfrac{dy}{dx}\)). We are differentiating \(f(x)\) wrt \(x\).

In integrals, it appears as a piece being added up (\(f(x)\,dx\)). We are integrating \(f(x)\) wrt \(x\).

L’Hôpital’s Rule

If \(\lim_{x \to c} f(x) = \lim_{x \to c} g(x) = 0\) or \(\pm \infty\), and \(\lim_{x \to c} \dfrac{\frac{d}{dx}f(x)}{\frac{d}{dx}g(x)}\) exists, then

\[\lim_{x \to c} \dfrac{f(x)}{g(x)} = \lim_{x \to c} \dfrac{\frac{d}{dx}f(x)}{\frac{d}{dx}g(x)}\]

Implicit Differentiation

We can use \[a = b \implies \dfrac{d}{dx}a = \dfrac{d}{dx}b\] to compute derivatives of relations (linked variables eg. the equation for a circle centered at the origin and of radius 5: \(x^2 + y^2 = 25\))

Or even to compute the derivative of \(ln(x)\):

\[\begin{aligned} y = ln(x) & \iff e^y = x \implies \dfrac{d}{dx} e^y = \dfrac{d}{dx} x \\ & \iff \dfrac{d}{dx} e^y = 1 \\ & \iff e^y\dfrac{d}{dx}y = 1 \\ & \iff \dfrac{d}{dx}y = \dfrac{1}{e^y} \\ & \iff \dfrac{d}{dx}y = \dfrac{1}{e^{ln(x)}} \\ & \iff \dfrac{d}{dx}y = \dfrac{1}{x} \end{aligned}\]

Proofs

Later!

Notation

• \(f(\cdot)\): a function
• \(\dfrac{dy}{dx}\): derivative of \(y\) with respect to \(x\)
• \(\dfrac{\partial y}{\partial x}\): partial derivative of \(y\) with respect to \(x\)
• \(\nabla_{\mathbf{x}} y\): gradient of \(y\) with respect to \(\mathbf{x}\)
• \(\mathbf{J}_f (\mathbf{x}), \dfrac{\partial \mathbf{y}}{\partial \mathbf{x}}\): Jacobian of \(\mathbf{y} = f(\mathbf{x})\) with respect to \(\mathbf{x}\)
• \(\int_a^b f(x) \;dx\): definite integral of \(f\) from \(a\) to \(b\) with respect to \(x\)
• \(\int f(x) \;dx\): indefinite integral of \(f\) with respect to \(x\)