Gaussian mean estimation HOMEWORK 1 – V3 Solution

Description

1. Gaussian mean estimation.

1.1. Optimal shrinkage factor [10pts]. Let X₁; X₂; :::; X_n 2 R^d be i.i.d. multivariate Gaussian random vectors, i.e., Xi N ( ; ²I). Denoting the sample mean estimator with ^ , ^{1 Pn} Xi,

n i=1

consider an estimator of the form ^s = 1		^: Find the optimal that minimizes the risk
	2
R(^s; ) = E[k^s k22].	k^k2

1.2. Generalizing Stein’s lemma [10pts]. Let X p (x) and g : R^d ! R^d where p (x) and g (x) are di erentiable w.r.t and x, and let E[g (X)] = ( ) for some function . Show that

1. E[r_x log p (X)g (X)^>] + E[r_xg (X)] = 0,

2. E[r log p (X)g (X)^>] + E[r g (X)] = r ( ).

1.3. Generalizing SURE [10pts]. Let X N ( ; ) where 2 R^d and 2 R^{d d}. If ^(X) 2 R^d is an estimator of the form X + g(X) where g : R^d ! R^d is di erentiable. De ne the functional

S(X; ^) = Tr( ) + 2 Tr( r_xg(X)) + kg(X)k²₂:

Then show that S(X; ^) is an unbiased estimator of the risk, i.e., E[k^(x) k²₂] = E[S(X; ^)].

2. Exponential families.

2.1. Second moment [10pts]. For a random variable X p (x) = exp(h ; (x)i ( )), let

E[ (X)] = . For 2 R^d, nd Tr(E[( (X) )( (X) )^>]) in terms of and rⁱ ( ) for i 0.

2.2. Score function [10pts]. Assume that X p (x) where p is not necessarily in the expo-nential family form. Denote the log-likelihood by ‘ (x) = log p (x), show that

1. E[r ‘ (X)] = 0.

2. E[r ‘ (X)r ‘ (X)^>] = E[r²‘ (X)] (Problem 1.2 may be helpful).

2.3. Maximum entropy principle [Bonus 2pts]. Assume that p(x) is a probability mass function of a discrete random variable taking values from a nite set X . Entropy of p is de ned as H(p) = ^P_x2X p(x) log p(x). For : X ! R^d, show that the maximum entropy distribution satisfying

E_p[ (X)] = 2 R^d is a member of exponential family. That is, show that the solution to

maximizeH(p) subject to: E_p[ (X)] = ;

p

is an exponential family. (Hint: Write the Lagrangian associated with the above optimization problem. Since X is nite, think of p(x) as a vector and maximize over it.)

1