Homework 1 Solution

Description

1. (25 points) Linear algebra refresher.

1. 1. (12 points) Let A be a square matrix, and further let AA^T = I.

1. 1. 1. (3 points) Construct a 2 2 example of A and derive the eigenvalues and eigen-vectors of this example. Show all work (i.e., do not use a computer’s eigenvalue decomposition capabilities). You may not use a diagonal matrix as your 2 2 example. What do you notice about the eigenvalues and eigenvectors?

1. 1. 2. (3 points) Show that A has eigenvalues with norm 1.

1. 1. 3. (3 points) Show that the eigenvectors of A corresponding to distinct eigenvalues are orthogonal.

1. 1. 4. (3 points) In words, describe what may happen to a vector x under the transfor-mation Ax.

1. 2. (8 points) Let A be a matrix.

1. 1. 1. (4 points) What is the relationship between the singular vectors of A and the eigenvectors of AA^T ? What about A^T A?

1. 1. 2. (4 points) What is the relationship between the singular values of A and the eigen-values of AA^T ? What about A^T A?

1. 3. (5 points) True or False. Partial credit on an incorrect solution may be awarded if you justify your answer.

1. 1. 1. Every linear operator in an n-dimensional vector space has n distinct eigenvalues.

1. 1. 2. A non-zero sum of two eigenvectors of a matrix A is an eigenvector.

1. 1. 3. If a matrix A has the positive semide nite property, i.e., x^T Ax 0 for all x, then its eigenvalues must be non-negative.

1. 1. 4. The rank of a matrix can exceed the number of non-zero eigenvalues.

1. 1. 5. A non-zero sum of two eigenvectors of a matrix A corresponding to the same eigenvalue is always an eigenvector.

2. (22 points) Probability refresher.

1. 1. (9 points) A jar of coins is equally populated with two types of coins. One is type \H50″ and comes up heads with probability 0:5. Another is type \H60″ and comes up heads with probability 0:6.

1. 1. 1. (3 points) You take one coin from the jar and ip it. It lands tails. What is the posterior probability that this is an H50 coin?

1. • 2. (3 points) You put the coin back, take another, and ip it 4 times. It lands T, H, H, H. How likely is the coin to be type H50?

1. • 2. (3 points) A new jar is now equally populated with coins of type H50, H55, and H60 (with probabilities of coming up heads 0:5, 0:55, and 0:6 respectively. You take one coin and ip it 10 times. It lands heads 9 times. How likely is the coin to be of each possible type?

2. (3 points) Consider a pregnancy test with the following statistics.

2. • If the woman is pregnant, the test returns \positive” (or 1, indicating the woman is pregnant) 99% of the time.

2. • If the woman is not pregnant, the test returns \positive” 10% of the time.

2. • At any given point in time, 99% of the female population is not pregnant.

What is the probability that a woman is pregnant given she received a positive test? The answer should make intuitive sense; given an explanation of the result that you nd.

3. (5 points) Let x₁; x₂; : : : ; x_n be identically distributed random variables. A random vector, x, is de ned as

	2	x1	3
x =	6 x...2		7
	6	x	7
	6	n	7
	4		5
What is E (Ax + b) in terms of E(x), given that A and b are deterministic?
(d) (5 points) Let
cov(x) = E (x		Ex)(x Ex)T
What is cov(Ax + b) in terms of cov(x), given that A				and b are deterministic?

3. (13 points) Multivariate derivatives.

3. 1. (2 points) Let x 2 Rⁿ, y 2 R^m, and A 2 R^{n m}. What is r_xx^T Ay?

3. 2. (2 points) What is r_yx^T Ay?

   3. (3 points) What is r_Ax^T Ay?

   4. (3 points) Let f = x^T Ax + b^T x. What is r_xf?

   5. (3 points) Let f = tr(AB). What is r_Af?

3. (10 points) Deriving least-squares with matrix derivatives.

In least-squares, we seek to estimate some multivariate output y via the model

y^ = Wx

In the training set we’re given paired data examples (x⁽ⁱ⁾; y⁽ⁱ⁾) from i = 1; : : : ; n. Least-squares is the following quadratic optimization problem:

Derive the optimal W.

Hint: you may nd the following derivatives useful:

^@tr(WA) _{= A}T @W

4. • ^{tr(WAWT )} _{= WA}T _{+ WA} @W

4. (30 points) Hello World in Jupyter.

Complete the Jupyter notebook linear regression.ipynb. Print out the Jupyter notebook and submit it to Gradescope.

3