Notation

Throughout this book, we adhere to the following notational conventions. Note that some of these symbols are placeholders, while others refer to specific objects. As a general rule of thumb, the indefinite article “a” often indicates that the symbol is a placeholder and that similarly formatted symbols can denote other objects of the same type. For example, “\(x\): a scalar” means that lowercased letters generally represent scalar values, but “\(\mathbb{Z}\): the set of integers” refers specifically to the symbol \(\mathbb{Z}\).

Numerical Objects

  • \(x\): a scalar

  • \(\mathbf{x}\): a vector

  • \(\mathbf{X}\): a matrix

  • \(\mathsf{X}\): a general tensor

  • \(\mathbf{I}\): the identity matrix (of some given dimension), i.e., a square matrix with \(1\) on all diagonal entries and \(0\) on all off-diagonals

  • \(x_i\), \([\mathbf{x}]_i\): the \(i^\textrm{th}\) element of vector \(\mathbf{x}\)

  • \(x_{ij}\), \(x_{i,j}\),\([\mathbf{X}]_{ij}\), \([\mathbf{X}]_{i,j}\): the element of matrix \(\mathbf{X}\) at row \(i\) and column \(j\).

Set Theory

  • \(\mathcal{X}\): a set

  • \(\mathbb{Z}\): the set of integers

  • \(\mathbb{Z}^+\): the set of positive integers

  • \(\mathbb{R}\): the set of real numbers

  • \(\mathbb{R}^n\): the set of \(n\)-dimensional vectors of real numbers

  • \(\mathbb{R}^{a\times b}\): The set of matrices of real numbers with \(a\) rows and \(b\) columns

  • \(|\mathcal{X}|\): cardinality (number of elements) of set \(\mathcal{X}\)

  • \(\mathcal{A}\cup\mathcal{B}\): union of sets \(\mathcal{A}\) and \(\mathcal{B}\)

  • \(\mathcal{A}\cap\mathcal{B}\): intersection of sets \(\mathcal{A}\) and \(\mathcal{B}\)

  • \(\mathcal{A}\setminus\mathcal{B}\): set subtraction of \(\mathcal{B}\) from \(\mathcal{A}\) (contains only those elements of \(\mathcal{A}\) that do not belong to \(\mathcal{B}\))

Functions and Operators

  • \(f(\cdot)\): a function

  • \(\log(\cdot)\): the natural logarithm (base \(e\))

  • \(\log_2(\cdot)\): logarithm to base \(2\)

  • \(\exp(\cdot)\): the exponential function

  • \(\mathbf{1}(\cdot)\): the indicator function; evaluates to \(1\) if the boolean argument is true, and \(0\) otherwise

  • \(\mathbf{1}_{\mathcal{X}}(z)\): the set-membership indicator function; evaluates to \(1\) if the element \(z\) belongs to the set \(\mathcal{X}\) and \(0\) otherwise

  • \(\mathbf{(\cdot)}^\top\): transpose of a vector or a matrix

  • \(\mathbf{X}^{-1}\): inverse of matrix \(\mathbf{X}\)

  • \(\odot\): Hadamard (elementwise) product

  • \([\cdot, \cdot]\): concatenation

  • \(\|\cdot\|_p\): \(\ell_p\) norm

  • \(\|\cdot\|\): \(\ell_2\) norm

  • \(\langle \mathbf{x}, \mathbf{y} \rangle\): inner (dot) product of vectors \(\mathbf{x}\) and \(\mathbf{y}\)

  • \(\sum\): summation over a collection of elements

  • \(\prod\): product over a collection of elements

  • \(\stackrel{\textrm{def}}{=}\): an equality asserted as a definition of the symbol on the left-hand side

Calculus

  • \(\frac{dy}{dx}\): derivative of \(y\) with respect to \(x\)

  • \(\frac{\partial y}{\partial x}\): partial derivative of \(y\) with respect to \(x\)

  • \(\nabla_{\mathbf{x}} y\): gradient of \(y\) with respect to \(\mathbf{x}\)

  • \(\int_a^b f(x) \;dx\): definite integral of \(f\) from \(a\) to \(b\) with respect to \(x\)

  • \(\int f(x) \;dx\): indefinite integral of \(f\) with respect to \(x\)

Probability and Information Theory

  • \(X\): a random variable

  • \(P\): a probability distribution

  • \(X \sim P\): the random variable \(X\) follows distribution \(P\)

  • \(P(X=x)\): the probability assigned to the event where random variable \(X\) takes value \(x\)

  • \(P(X \mid Y)\): the conditional probability distribution of \(X\) given \(Y\)

  • \(p(\cdot)\): a probability density function (PDF) associated with distribution \(P\)

  • \({E}[X]\): expectation of a random variable \(X\)

  • \(X \perp Y\): random variables \(X\) and \(Y\) are independent

  • \(X \perp Y \mid Z\): random variables \(X\) and \(Y\) are conditionally independent given \(Z\)

  • \(\sigma_X\): standard deviation of random variable \(X\)

  • \(\textrm{Var}(X)\): variance of random variable \(X\), equal to \(\sigma^2_X\)

  • \(\textrm{Cov}(X, Y)\): covariance of random variables \(X\) and \(Y\)

  • \(\rho(X, Y)\): the Pearson correlation coefficient between \(X\) and \(Y\), equals \(\frac{\textrm{Cov}(X, Y)}{\sigma_X \sigma_Y}\)

  • \(H(X)\): entropy of random variable \(X\)

  • \(D_{\textrm{KL}}(P\|Q)\): the KL-divergence (or relative entropy) from distribution \(Q\) to distribution \(P\)

Discussions