CS559-B HW1

Description

Problem 1 (5pt): Provide an intuitive example to show that P (A|B) and P (B|A) are in general not the same. Provide matrix examples to show AB ̸= BA. (No math derivation is needed).

Problem 2 (10pt): Independence and un-correlation

1. (5pt) Suppose X and Y are two continuous random variables, show that if X and Y are independent, then they are uncorrelated.

2. (5pt) Suppose X and Y are uncorrelated, can we conclude X and Y are independent? If so, prove it, otherwise, give one counterexample. (Hint: consider X ∼ U nif orm[−1, 1] and Y = X²)

Problem 2 (15pt): [Minimum Error Rate Decision] Let ω_max(x) be state of nature for which P (ω_max|x) ≥ P (ω_i|x) for all i = 1, . . . , c.

1. Show that P (ω_max|x) ≥ ¹_c

2. Show that for minimum-error-rate decision rule, the average probability of error is given by

P (error) = 1 − P (ω_max|x)p(x)dx

(3) Show that P (error) ≤ ^c−_c¹

Problem 4 (10pt): [Likelihood Ratio] Suppose we consider two category classification, the class conditionals are assumed to be Gaussian, i.e., p(x|ω₁) = N(4, 1) and p(x|ω₂) = N(8, 1), based on prior knowledge, we have P (ω₂) = ¹₄ . We do not penalize for correct classification, while for misclassification, we put 1 unit penalty for misclassifying ω₁ to ω₂ and put 3 unit for misclassifying ω₂ to ω₁. Derive the bayesian decision rule using likelihood ratio.

Problem 5 (15pt): [Minimum Risk, Reject Option] In many machine learning applications, one has the option either to assign the pattern to one of c classes, or to reject it as being unrecognizable. If the cost for reject is not too high, rejection may be a desirable action. Let

0, i = j and i, j = 1, . . . , c

λ(αi|ωj) = λ_r, i = c + 1

λ_s, otherwise

where λ_r is the loss incurred for choosing the (c + 1)-th action, rejection, and λ_s is the loss incurred for making any substitution error.

1. (5pt) Derive the decision rule with minimum risk.

2. (5pt) What happens if λ_r = 0?

3. (5pt) What happens if λ_r > λ_s?

Problem 6 (25pt): [Maximum Likelihood Estimation (MLE)] A general representation of a exponential family is given by the following probability density:

p(x|η) = h(x) exp{η^T T (x) − A(η)}

• η is natural parameter.

• h(x) is the base density which ensures x is in right space.

• T (x) is the sufficient statistics.

• A(η) is the log normalizer which is determined by T (x) and h(x).

• exp(.) represents the exponential function.

• 1. (5pt) Write down the expression of A(η) in terms of T (x) and h(x).

• 2. (10pt) Show that _∂η^∂ A(η) = E_ηT (x) where E_η(.) is the expectation w.r.t p(x|η).

• 3. (10pt) Suppose we have n i.i.d samples x₁, x₂, . . . , x_n, derive the maximum likelihood esti-mator for η. (You may use the results from part(b) to obtain your final answer)

Problem 7 (20pt): [Logistic Regression, MLE] In this problem, you need to use MLE to derive and build a logistic regression classifier (suppose the target/response y ∈ {0, 1}):

1. (5pt) Suppose the classifier is y = x^T θ, where θ contains the weight as well as bias parame-ters. The log-likelihood function is LL(θ), what is ^∂LL_∂θ^(θ) ?

2. (15pt) Write the codes to build and train the classifier on Diabetes dataset (attached in Canvas). The Diabetes dataset contains 768 samples with 9 features for 2 outcomes. To simplify the problem, we only consider: Glucose and BMI as our features. Based on the simplified settings, train the model using gradient descent. Please show the classification results. (Note that (1) you could split the Diabetes dataset into train/test set. (2) You could visualize the results by showing the trained classifier overlaid on the train/test data. (3) You could tune several hyperparameters, e.g., learning rate, weight initialization method etc, to see their effects. (3) you can not use the package to directly train the model (e.g., sklearn.linear model.LogisticRegression)).

2