Characteristic function (probability theory)

From Wikipedia, the free encyclopedia

In probability theory and statistics, the characteristic function of any random variable completely defines its probability distribution. Thus it provides the basis of an alternative route to analytical results compared with working directly with probability density functions or cumulative distribution functions. There are particularly simple results for the characteristic functions of distributions defined by the weighted sums of random variables.

In addition to univariate distributions, characteristic functions can be defined for vector- or matrix-valued random variables, and can even be extended to more generic cases.

The characteristic function always exists when treated as a function of a real-valued argument, unlike the moment-generating function. There are relations between the behaviour of the characteristic function of a distribution and properties of the distribution, such as the existence of moments and the existence of a density function.

[edit] Introduction

The characteristic function provides an alternative way for describing a random variable. Similarly to the cumulative distribution function

$F_X(x)=\operatorname{E}\,\mathbf{1}_{\{X\leq x\}}$

which completely determines behavior and properties of the probability distribution of the random variable X, the characteristic function

$\varphi_X(t)=\operatorname{E}\,e^{itX}$

also completely determines behavior and properties of the probability distribution of the random variable X. The two approaches are equivalent in the sense that knowledge of one of the functions can always be used find the other one, yet they both provide different insight for understanding the features of our random variable. However, in particular cases, there can be differences in whether these functions can be repesented as expressions involving simple standard functions.

If a random variable admits a density function, then the characteristic function is its dual, in the sense that each of them is a Fourier transform of the other. If a random variable has a moment-generating function, then the characteristic function can be extended to the complex domain so that

$\varphi_X(-it)=M_X(t).$ ^[1]

Note however that the characteristic function of a distribution always exists, even when the probability density function or moment-generating function do not.

The characteristic function approach is particularly useful in analysis of linear combinations of independent random variables. Another important application is to the theory of the decomposability of random variables.

[edit] Definition

The characteristic function of any random variable is a Fourier–Stieltjes transform of its distribution function. For a scalar random variable X the characteristic function is defined as

$\varphi_X:\mathbb{R}\to\mathbb{C}; \quad \varphi_X(t) = \operatorname{E}\,e^{itX} = \int_{-\infty}^\infty e^{itx}\,dF_X(x) \qquad \left( = \int_{-\infty}^\infty e^{itx} f_X(x)\,dx \right),$

where t is a real number, i is the imaginary unit, E denotes expected value, F_X is the cumulative distribution function of X, and integral is Riemann–Stieltjes. If random variable X has a probability density function $f X$ , then the characteristic function is its Fourier transform (Billingsley 1995), and the last formula in parentheses is valid. It should be noted though, that this convention for the constants appearing in the definition of the characteristic function differs from the usual convention for the Fourier transform(Pinsky 2002). Other ways of defining the characteristic function are also used: some authors also define $\varphi_X(t)=\operatorname{E}\,e^{-2\pi itX}$ (Bochner 1955), which is essentially a different change of parameter.

Other notation may be encountered in published literature: $\hat p$ as the characteristic function for aprobability measure p, or $\hat f$ as the characteristic function corresponding to a density ƒ.

As defined above, the argument of the characteristic function is treated as a real number: however, certain aspects of the theory of characteristic functions are advanced by extending the definition into the complex plane by analytical continuation, in cases where this is available.^[2]

The notion of characteristic functions can be generalized to include random variables defined on an arbitrary Hilbert space H with inner product $\langle,\rangle$

$\varphi_X:H\to\mathbb{C}; \quad \varphi_X(t) = \operatorname{E}\,e^{i \langle t,X\rangle} = \int_H e^{i \langle t,x\rangle}\,F_X(dx).$

For specific examples see section Multivariate characteristic functions below.

[edit] Examples

Distribution	Characteristic function φ(t)
Degenerate δ_a	$\, e^{ita}$
Binomial B(n, p)	$\, (1-p+pe^{it})^n$
Poisson Pois(λ)	$\, e^{\lambda(e^{it}-1)}$
Uniform U(a, b)	$\, \frac{e^{itb} - e^{ita}}{it(b-a)}$
Laplace L(μ, b)	$\, \frac{e^{it\mu}}{1 + b^2t^2}$
Normal N(μ, σ²)	$\, e^{it\mu - \frac{1}{2}\sigma^2t^2}$
Chi-square χ²_k	$\, (1 - 2it)^{-k/2}$
Cauchy Cauchy(μ, θ)	$\, e^{it\mu -\theta\|t\|}$
Gamma Γ(k, θ)	$\, (1 - it\theta)^{-k}$
Exponential Exp(λ)	$\, (1 - it\lambda^{-1})^{-1}$
Multivariate normal N(μ, Σ)	$\, e^{it'\mu - \frac{1}{2}t'\Sigma t}$

[edit] Properties

The characteristic function of a random variable always exists, because it is an integral of a bounded function over a space whose measure is finite.
A characteristic function is uniformly continuous on the entire space
It is non-vanishing in a region around zero: $\varphi(0)=1$
It is bounded: $|\varphi(t)|\leq 1$
It is Hermitian: $\varphi(-t) = \overline{\varphi(t)}$ . In particular, the characteristic function of a symmetric random variable is real-valued and even.
There is a bijection between distribution functions and characteristic functions. That is, for any two random variables $X 1$ , $X 2$

$F_{X_1}=F_{X_2}\ \Leftrightarrow\ \varphi_{X_1}=\varphi_{X_2}$

If random variable X has moments up to k-th order, then characteristic function $\varphi_X$ is k times continuously differentiable on entire real line.
If characteristic function $\varphi_X$ has k-th derivative at zero, then random variable X has all moments up to k if k is even, but only up to k-1 if k is odd.^[3]

[edit] Continuity

The bijection stated above between probability distributions and characteristic functions is continuous. That is, whenever a sequence of distribution functions ${F j (x)}$ converges (weakly) to some distribution $F (x)$ , the corresponding sequence of characteristic functions {φ_j(t)} will also converge, and the limit φ(t) will correspond to the characteristic function of law F. More formally, this is stated as

Lévy's continuity theorem. A sequence

{X j}

of n-variate random variables converges in distribution to random variable

X

if and only if the sequence $\{\varphi_{X_j}\}$ converges pointwise to a function $\varphi$ which is continuous at the origin. Then $\varphi$ is the characteristic function of

X

.^[4]

Main article: Lévy continuity theorem

This theorem is frequently used to prove the Law of Large Numbers, and the Central Limit Theorem.

[edit] Inversion formulas

Since there is a one-to-one correspondence between cumulative distribution functions and characteristic functions, it is always possible to find one of these functions if we know the other one. The formula in definition of characteristic function allows us to compute φ when we know the distribution function F (or density f). If, on the other hand, we know the characteristic function φ and want to find the corresponding distribution function, then one of the following inversion theorems can be used.

Theorem. If characteristic function φ_X is integrable, then F_X is absolutely continuous, and therefore X has probability density function (in multivariate case it is understood as the Radon-Nikodym derivative of F_X with respect to the Lebesgue measure λ) given by

$f_X(x) = F_X'(x) = \frac{1}{2\pi}\int_{-\infty}^{\infty} e^{-itx}\varphi_X(t)dt,$ if X is scalar

$f_X(x) = \frac{dF_X}{d\lambda}(x) = \frac{1}{(2\pi)^n} \int_{\mathbb{R}^n} e^{-i(t\cdot x)}\varphi_X(t)\lambda(dt),$ if X is a vector random variable.

Theorem (Lévy). If φ_X is characteristic function of distribution function F_X, two points a<b are such that ${x | a < x < b}$ is a continuity set of F_X (in univariate case this condition is equivalent to continuity of F_X at points a and b), then

$F_X(b) - F_X(a) = \frac{1} {2\pi} \lim_{T \to \infty} \int_{-T}^{+T} \frac{e^{-ita} - e^{-itb}} {it}\, \varphi_X(t)\, dt,$ if X is scalar

$F_X\big(\{a<x<b\}\big) = \frac{1}{(2\pi)^n} \lim_{T_1\to\infty}\cdots\lim_{T_n\to\infty} \int\limits_{\{-T\leq t\leq T\}} \prod_{k=1}^n\left(\frac{e^{-it_ka_k}-e^{-it_kb_k}}{it_k}\right)\varphi_X(t)\lambda(dt)$ , if X is a vector random variable.

Theorem. If a is a point of (potential) discontinuity of F_X then

$F_X(a) - F_X(a-0) = \lim_{T\to\infty}\frac{1}{2T}\int_{-T}^{+T}e^{-ita}\varphi_X(t)dt$ , when X is a scalar random variable

$F_X(\{a\}) = \lim_{T_1\to\infty}\cdots\lim_{T_n\to\infty} \left(\prod_{k=1}^n\frac{1}{2T_k}\right) \int\limits_{\{-T\leq t\leq T\}} e^{-i(t\cdot x)}\varphi_X(t)\lambda(dt)$ , when X is a vector random variable.

Theorem (Gil-Pelaez).^[5] For a univariate random variable X, if x is a continuity point of F_X then

$F_X(x) = \frac{1}{2} - \frac{1}{\pi}\int_{0}^{\infty} \frac{\operatorname{Im}[e^{-itx}\varphi_X(t)]}{t}dt.$

[edit] Criteria for characteristic functions

It is well-known that any non-decreasing càdlàg function F with limits $F(-\infty)=0$ , $F(+\infty)=1$ corresponds to a cumulative distribution function of some random variable.

Now we are also interested in finding similar simple criteria for when given function φ could be the characteristic function of some random variable. Central result here is Bochner's theorem, although its usefulness is limited because the main condition of the theorem, non-negative definiteness, is very hard to verify. Other theorems also exist, such as Khinchine's, Mathias's, or Cramér's, although their application is just as difficult. Pólya theorem, on the other hand, provides a very simple convexity condition which is sufficient but not necessary. Characteristic functions which satisfy this condition are called Pólya-type.^[6]

Bochner's theorem. An arbitrary function $\varphi:\mathbb{R}^n\to\mathbb{C}$ is a characteristic function of some random variable if and only if φ is nonnegative definite, continuous at the origin, and if φ(0)=1.
Main article: Bochner's theorem
Khinchine's criterion. An absolutely continuous complex-valued function φ equal to 1 at the origin is a characteristic function if and only if it admits the representation

$\varphi(t) = \int_{-\infty}^\infty g(t+\theta)\overline{g(\theta)} d\theta$
Mathias' theorem. A real, even, continuous, absolutely integrable function φ equal to 1 at the origin is a characteristic function if and only if

$(-1)^n \int_{-\infty}^\infty \varphi(pt)e^{-t^2/2}H_{2n}(t)dt \geq 0$

for all n=0,1,2,…, and all p>0. Here $H 2 n$ denotes Hermite polynomial of degree 2n.
A convex linear combination $\textstyle \sum_n a_n\varphi_n(t)$ (with $\textstyle a_n\geq0,\ \sum_n a_n=1$ ) of finite or countable number of characteristic functions is also a characteristic function.
Product $\textstyle \prod_n \varphi_n(t)$ of finite or countable number of characteristic functions is also a characteristic function.
If φ is a characteristic function, then so are $\overline\varphi,\ \operatorname{Re}\,\varphi,\ \operatorname{Im}\,\varphi,\ |\varphi|^2.$
Pólya theorem. If φ is real-valued continuous function which satisfies conditions
1. φ(0)=1,
2. φ is even,
3. φ is convex for t>0,
4. φ(∞)=0,
then φ(t) is the characteristic function of an absolutely continuous symmetric distribution.

[edit] Uses

Because of the continuity theorem, characteristic functions are used in the most frequently seen proof of the central limit theorem. The main trick involved in making calculations with a characteristic function is recognizing the function as the characteristic function of a particular distribution.

[edit] Basic manipulations of distributions

Characteristic functions are particularly useful for dealing with functions of independent random variables. For example, if X₁, X₂, ..., X_n is a sequence of independent (and not necessarily identically distributed) random variables, and

$S_n = \sum_{i=1}^n a_i X_i,\,\!$

where the a_i are constants, then the characteristic function for S_n is given by

$\varphi_{S_n}(t)=\varphi_{X_1}(a_1t)\varphi_{X_2}(a_2t)\cdots \varphi_{X_n}(a_nt). \,\!$

In particular, $\varphi_{X+Y}(t) = \varphi_X(t)\varphi_Y(t)$ . To see this, write out the definition of characteristic function:

$\varphi_{X+Y}(t)=E\left(e^{it(X+Y)}\right)=E\left(e^{itX}e^{itY}\right)=E\left(e^{itX}\right)E\left(e^{itY}\right)=\varphi_X(t) \varphi_Y(t)$ .

Observe that the independence of $X$ and $Y$ is required to establish the equality of the third and fourth expressions.

Another special case of interest is when $a i = 1 / n$ and then $S n$ is the sample mean. In this case, writing $\overline{X}$ for the mean,

$\varphi_{\overline{X}}(t)=\left(\varphi_X(t/n)\right)^n.$

[edit] Moments

Characteristic functions can also be used to find moments of a random variable. Provided that the n^th moment exists, characteristic function can be differentiated n times and

$\operatorname{E}\left(X^n\right) = i^{-n}\, \varphi_X^{(n)}(0) = i^{-n}\, \left[\frac{d^n}{dt^n} \varphi_X(t)\right]_{t=0}. \,\!$

For example, suppose $X$ has a standard Cauchy distribution. Then $\varphi_X(t)=e^{-|t|}$ . See how this is not differentiable at $t = 0$ , showing that the Cauchy distribution has no expectation. Also see that the characteristic function of the sample mean $\overline{X}$ of $n$ independent observations has characteristic function $\varphi_{\overline{X}}(t)=(e^{-|t|/n})^n=e^{-|t|}$ , using the result from the previous section. This is the characteristic function of the standard Cauchy distribution: thus, the sample mean has the same distribution as the population itself.

The logarithm of a characteristic function is a cumulant generating function, which is useful for finding cumulants; note that some instead define the cumulant generating function as the logarithm of the moment-generating function, and call the logarithm of the characteristic function the second cumulant generating function.

[edit] Data analysis

Characteristic functions can be used as part of procedures for fitting probability distributions to samples of data. Cases where this provides a practicable option compared to other possibilities include fiiting the stable distribution since closed form expressions from the density are not available which makes implementation of maximum likelihood estimation difficult. Estimation procedures are available which match the theoretical characteristic function to the empirical characteristic function, calculated from the data. Paulson et al. (1975) and Heathcote (1977) provide some theoretical background for such an estimation procedure.

[edit] Example

The Gamma distribution with scale parameter θ and a shape parameter k has the characteristic function

$(1 - \theta\,i\,t)^{-k}.$

Now suppose that we have

$X ~\sim \Gamma(k_1,\theta) \mbox{ and } Y \sim \Gamma(k_2,\theta)$

with X and Y independent from each other, and we wish to know what the distribution of X + Y is. The characteristic functions are

$\varphi_X(t)=(1 - \theta\,i\,t)^{-k_1},\,\qquad \varphi_Y(t)=(1 - \theta\,i\,t)^{-k_2}$

which by independence and the basic properties of characteristic function leads to

$\varphi_{X+Y}(t)=\varphi_X(t)\varphi_Y(t)=(1 - \theta\,i\,t)^{-k_1}(1 - \theta\,i\,t)^{-k_2}=\left(1 - \theta\,i\,t\right)^{-(k_1+k_2)}.$

This is the characteristic function of the gamma distribution scale parameter θ and shape parameter k₁ + k₂, and we therefore conclude

$X+Y \sim \Gamma(k_1+k_2,\theta) \,$

The result can be expanded to n independent gamma distributed random variables with the same scale parameter and we get

$\forall i \in \{1,\ldots, n\} : X_i \sim \Gamma(k_i,\theta) \qquad \Rightarrow \qquad \sum_{i=1}^n X_i \sim \Gamma\left(\sum_{i=1}^nk_i,\theta\right).$

[edit] Multivariate characteristic functions

[edit] Vector-valued random variables

If $X$ is an n-dimensional multivariate random variable, then its characteristic function is defined as

$\varphi_X(t)=\operatorname{E}\,e^{it'X},$

where $t\in\mathbb{R}^n$ and $'$ denotes transposition.

[edit] Matrix-valued random variables

If $X$ is an m×n matrix-valued random variable, then its characteristic function is

$\varphi_X(T)=\operatorname{E}\,e^{i\,\operatorname{tr}(T'\!X)},$

where T is also an m×n matrix, tr(·) is the trace operator and matrix multiplication (of T' and X) is used. The order of matrix multiplication is immaterial: although X'T≠T'X, but by the properties of trace tr(X'T)=tr(T'X). Note that trace operator defines an inner product on the space of m×n matrices.

Examples of matrix-valued random variables include Wishart distribution, Matrix normal distribution, and others.

[edit] Stochastic processes

If $X (t)$ is a stochastic process, then characteristic function becomes a characteristic functional. It is defined for a non-random test function $f (t)$ such that integral $\int f(t)X(t)dt \equiv \langle f,X\rangle$ converges for almost all realizations of X:^[7]

$\varphi_X(f)=\operatorname{E}\!\left[ \exp\Big(i\!\int\!f(t)X(t)dt\Big) \right].$

[edit] Related concepts

Related concepts include the moment-generating function and the probability-generating function. The characteristic function exists for all probability distributions. However this is not the case for moment generating function.

The characteristic function is closely related to the Fourier transform: the characteristic function of a probability density function $p (x)$ is the complex conjugate of the continuous Fourier transform of $p (x)$ (according to the usual convention; see continuous Fourier transform – alternative forms).

$\varphi_X(t) = \langle e^{itX} \rangle = \int_{-\infty}^{\infty} e^{itx}p(x)\, dx = \overline{\left( \int_{-\infty}^{\infty} e^{-itx}p(x)\, dx \right)} = \overline{P(t)},$

where $P (t)$ denotes the continuous Fourier transform of the probability density function $p (x)$ . Likewise, $p (x)$ may be recovered from $\varphi_X(t)$ through the inverse Fourier transform:

$p(x) = \frac{1}{2\pi} \int_{-\infty}^{\infty} e^{itx} P(t)\, dt = \frac{1}{2\pi} \int_{-\infty}^{\infty} e^{itx} \overline{\varphi_X(t)}\, dt.$

Indeed, even when the random variable does not have a density, the characteristic function may be seen as the Fourier transform of the measure corresponding to the random variable.

[edit] Notes

^ Lukacs (1970) p. 196
^ Lukacs (1970), Chapter 7
^ Lukacs (1970), Corollary 1 to Theorem 2.3.1
^ Cuppens (1975), Th.2.6.9
^ Wendel, J.G. (1961)
^ Lukacs (1970), p.84
^ Sobczyk (2001) p.20

[edit] References

Billingsley, Patrick (1995), Probability and Measure (Third ed.), John Wiley and Sons, ISBN 0-471-00710-2
Bisgaard, T. M., Sasvári, Z. (2000) Characteristic Functions and Moment Sequences, Nova Science
Bochner, Salomon (1955), Harmonic Analysis and the Theory of Probability, University of California Press
Cuppens, R. (1975) Decomposition of Multivariate Probabilities, Academic Press
Heathcote, C.R. (1977) The integrated squared error estimation of parameters. Biometrika, 64(2), 255–264
Lukacs E. (1970) Characteristic Functions. Griffin, London. pp. 350
Paulson, A.S., Holcomb, E.W., Leitch, R.A. (1975) The estimation of the parameters of the stable laws. Biometrika, 62, 163–170
Pinsky, Mark (2002), Introduction to Fourier Analysis and Wavelets, Brooks/Cole, ISBN 0-534-37660-6
Sobczyk, Kazimierz (2001). Stochastic Differential Equations. Kluwer Academic Publishers. ISBN 9781402003455.
Wendel, J.G. (1961). The Non-Absolute Convergence of Gil-Pelaez' Inversion Integral. The Annals of Mathematical Statistics, Vol. 32, No. 1, pp. 338–339

[0] Lukacs (1970) p. 196

[1] Lukacs (1970), Chapter 7

[2] Lukacs (1970), Corollary 1 to Theorem 2.3.1

[3] Cuppens (1975), Th.2.6.9

[4] Wendel, J.G. (1961)

[5] Lukacs (1970), p.84

[6] Sobczyk (2001) p.20

[1]

[2]

[3]

[4]

[5]

[6]

[7]