Summary
Discrete Random Variable Expected Value
Definition
Let be a discrete random variable. Then the expected value of is
Note
This can also be called the weighted average.
The expected value (expectation) is well-defined only when the sum is absolutely convergent, i.e. .
Example (Direct Computation)
. .
Example (Manipulation of Sums)
Example (Derivatives of Sums)
(see geometric RVs). . Let . Then:
Recall the geometric series for . Differentiating both sides with respect to :
Therefore .
Example (Taylor Expansion)
(see Poisson RVs). Recall for .
where we substituted and recognized the Taylor series for .
Continuous Random Variable Expected Value
Definition
If is a continuous random variable with probability density function , then the expected value of is defined as
under the assumption
Example
.
Example
Compute when (see exponential RVs).
Example
. . Substitute :
Example
Let , that is a continuous RV with p.d.f.
This is not finite when computed with . This is because the expression is asymptotic to which is an infinite integral.
By this example, if then is not defined. Also, .
Question
How can we compute where ?
Theorem
Let be discrete random variables with joint p.m.f. . Set where . Then
Proof
Note that and .
Since every tuple maps to exactly one , the double sum covers each tuple exactly once:
Theorem
Let be a continuous random vector with joint p.d.f. . Set where . Then
Proof
Analogous to the discrete case. By definition, . Writing as a marginal of the joint density:
Replacing with and collapsing the iterated integral (each point in maps to exactly one ):
Linearity of Expectation
Theorem
Let be random variables (discrete or continuous). Then
Proof
We will prove the discrete case for simplicity. By the previous theorem with :
Distributing the sum:
Each term equals (again by the previous theorem, with ):
Example
. Recall where (see Bernoulli RVs). Hence .
Example
Compute where (see gamma RVs).
The Method of Indicator Functions
Definition
For an event , we say the indicator function on is the random variable defined by
Notice that where . Additionally, . Finally, if , then by linearity.
Example
Suppose we have letters and envelopes. What is the expected number of letters in the correct envelope? , where . By linearity:
Example
Consider balls dropped at random in boxes. . Compute .
Let . .
Example
Toss times a -coin. An -run is a sequence of consecutive heads. .
. . . .
Notice that in these examples we are abusing linearity of expectation in a very useful way.
Conditional Expectation
Theorem
The tail integral formula for expectation gives the following. Let be a non-negative random variable ().
Proof
Note that because almost surely. .
Corollary
The discrete version is as follows. If is non-negative discrete random variable,
Definition
If is a random vector, the mean of a random vector is
Example
Let be a matrix, and . Define . Compute
Definition
Let be two random variables that are discrete. The conditional expectation of given is
where is the conditional p.m.f.
Definition
If are continuous random variables, the conditional expectation is
using the conditional p.d.f.
Example
. .
In this example, we demonstrated that it is actually quite common for the expectation to have zero probability, especially with discrete random variables!
Conditional Expectation as a Random Variable
This concept is very confusing, so do not worry if it seems nonsensical.
Definition
The conditional expectation of given is
This is a new random variable.
In the previous example, .
The moral is that is the best approximation for using only information in .
Theorem
Proof
where