There are various concepts in statistics that are quite often used in many other fields of study in real life. Mean, median, mode, standard deviation and variance are few common terminologies which are being utilized almost allover where ever large numerical data is studied. There are more uncommon, but very important concepts such as: covariance, correlation, covariance matrix etc. In this page, we are going to learn about covariance matrix. It is a matrix that uses variance and covariance as its elements. Covariance matrices are utilized in higher-level mathematics and fields like: engineering, computer science, physics, financial economics etc.

## What is Covariance Matrix

Covariance matrix is the matrix that has covariances as its elements. It has covariances between i$^{th}$ and j$^{th}$ elements of random variables of a random vector.
Covariance matrix is also known as variance-covariance matrix or dispersion matrix. These matrices are symmetric in nature, i.e. on interchanging the elements of rows by those of columns, same matrix is obtained. Covariance matrix of a random vector $X$ is denoted by $V[X]$ or $C[X]$.

$V[X]$ = $E$ $[(X_{i} - \mu_{i}) (X_{j} - \mu_{j})]$

Where, $\mu_{i}$ and $\mu_{j}$ are the expected values of vector $X$ at i$^{th}$ and j$^{th}$ position respectively.
These matrices generally has variances at diagonal elements and covariances as non-diagonal elements.

## Variance Covariance Matrix

Variance-covariance matrices, as the name suggests, do have variances and covariances as their elements. In this matrix, all the variances occur along the diagonal elements and all covariances occur along the non-diagonal elements. Variance-covariance matrices are the special way to represent the data and perform various calculations on it.

$A$ 3 x 3 order variance-covariance matrix for the random variable $\vec{x}$ can be represented in the following way :

$Cov$ ($\vec{x}$) = $\begin{bmatrix} var[X_{1}] & cov[X_{1},X_{2}] & cov[X_{1},X_{3}]\\ cov[X_{2},X_{1}] & var[X_{2}] & cov[X_{2},X_{3}]\\ cov[X_{3},X_{1}] & cov[X_{3},X_{2}] & var[X_{3}] \end{bmatrix}$

## Properties of Covariance Matrix

A covariance matrix possesses a number of properties. Few important of those are illustrated below :
Let us consider that $X$ and $Y$ are two random vectors. Also, $A$ and $B$ are two random matrices.

1) Covariance matrix is a symmetric matrix which means that even if rows are interchanged by columns and columns by row,
then also the matrix looks exactly similar to previous one.
i.e. $C[X]$ = $(C[X])$ $^{T}$

2) Covariance matrix is positive semi-definite i.e:
For every real number a in n-dimensional space:
$E[(X - \mu)^{T}\ a]^{2} \geq\ 0$

3) $C [X, Y]$ = $C [Y, X]$ $^{T}$

4) $var (X + Y)$ = $var (X)$ + $cov (X, Y)$ + $cov (Y, X)$ + $var(Y)$

## Sample Covariance Matrix

When we talk about covariance matrix, we generally mean to refer it to sample covariance matrix. In practical applications covariance matrix should be estimated from the data, and usually we estimate the covariance matrix using the data samples. In other words we can say that, the covariance matrix belonging to a sample taken out from population, is termed as sample covariance matrix. The sample covariance matrix is the maximum-likelihood estimate of the true covariance matrix.
The general form of a sample covariance matrix is defined as a $K \times K$ matrix, such that
$C$ = $c_{jk}$
Where,
$c_{jk}$=$\frac{1}{n-1}$ $\sum_{i=1}^{n}(x_{ij}- \bar{x_{j}})(x_{ik}- \bar{x_{k}})$
$c_{jk}$ is the sample covariance between $j^{th}$ and $k^{th}$ elements.

## Inverse Covariance Matrix

The covariance matrix and its inverse can be used in the estimation of a system's transfer function. However, the technique is only useful for symmetric covariance matrices.
A covariance matrix may also be denoted by $\sum$ and it can also be expressed in the following way:

$\sum =\begin{bmatrix} E(X_{1}-\mu_{1})(X_{1}-\mu_{1}) & ... & E(X_{1}-\mu_{1})(X_{n}-\mu_{n})\\ E(X_{2}-\mu_{2})(X_{1}-\mu_{1}) & ... & E(X_{2}-\mu_{2})(X_{n}-\mu_{n})\\ ... & ... & ...\\ E(X_{n}-\mu_{n})(X_{1}-\mu_{1}) & ... & E(X_{n}-\mu_{n})(X_{n}-\mu_{n}) \end{bmatrix}$
The inverse matrix of this covariance matrix is denoted by $\sum^{-1}$ and is known as inverse covariance matrix. It is being used quite frequently in many statistical estimations and researches as well.

Lets analyze the properties of the estimators by finding a closed a closed-form expression for the inverse of the covariance matrix based on the covariance matrix model i.e.
$A_k$ = E$(x_k (m) x_k^H (m))$

= $Z_k T_k Z_k^H + S_k$   ....(1)

We will use this model in the following. A variation of the matrix inversion lemma states that for matrices $M$, $N$, $P$, $Q$

$(M + NPQ)$ $^{-1}$ = $M^{-1}$ - $M^{-1}$ $N(P^{-1}$ + $QM^{-1}N)$ $^{-1}$ $QM^{-1}$

$A_k^{-1}$ = $(Z_k$ $T_k$ $Z_k^H$ + $S_k$)$^{-1}$

= $S_k^{-1}$ - $S_k^{-1}$ $Z_k$ $(T_k^{-1}$ + $Z_k^H$ $S_k^{-1}$ $Z_k$)$^{-1}$ $Z_k^H$ $S_k^{-1}$   ............(2)
Provided that the respective matrix inverses exist. It can seen that equation (2) is of the same form as equation (1).

## Covariance Matrix Example

Consider an example in which covariances are listed as follows:
 Weight Hours Height Life Span Weight 1 0.34 - 1.1 2.23 Hours 0.34 1 - 0.6 3.9 Height - 1.1 - 0.6 1 1.2 Life Span 2.23 3.9 1.2 1
If we have a look closely that the covariances corresponding to weight - weight, hours - hours, height - height and life span - life span are all 1) Also, the covariances corresponding to hours - weight and weight - hours are same. Similarly, covariances between weight - life span and life span - weight are same and so on.From above, following covariance matrix is obtained. We can easily observe that it is a symmetric matrix:

$\begin{bmatrix} 1 & 0.34 & -1.1 &2.23 \\ 0.34 & 1 & -0.6 & 3.9 \\ -1.1 & -0.6 & 1 & 1.2\\ 2.23 & 3.9 & 1.2 & 1 \end{bmatrix}$