Description
Note: All coding problems to be submited with Github Link. Do not Upload the files/folder. Use git commands only.
Note: this is the distribution of questions:
-
Question 1 to Question6: Required for everyone.
-
Question 7 – Question 8: Required only for Graduate Students
Problem 1 (5 points)
Accuracy is probably the most commonly used metric in classification problems.
Which statement below is True?:
-
Accuracy is very easy to interpret and is aligned with what you want to measure.
-
Accuracy is non-differentiable and cannot be used for direct optimization via gradient descent.
-
We want accuracy to count every error as equal.
-
If the number of samples in each category is comparable, we cannot make any mistake on the less populated category.
Problem 2 (10 points)
You are working as a Machine Learning Engineer in Metflix Inc. You are building a model to classify users who watch a lot of movies in Ultiverse. What metrics will you choose to evaluate your model?
Problem 3 (10 points)
Which method is used involved in numerical optimization of an appropriate se-lection of model criterion? How do you define the error of such estimator?
We covered Automatic Differentiation in class. Consider the following function:
-
-
-
-
-
f (x, y, z) =
1
(x1x2 sin(x3) + expx1x2 )
(1)
3
-
-
-
-
-
Draw a computation graph for this function.
-
provide the computation trace of the function
-
provide the forward AD trace for x1 = 1 , x2 = 0, x3 = 0
-
provide the reverse mode AD trace for x = 1 , y = 0, z = 0
List the detailed computation steps (the trace) for forward and backward mode of AD. Provide your answers the same way we did in class, by using notations like:
-
to get you started:
v−2, v−1, v0, v1, v2
(2)
∂v−2
v
−2
= 1, v˙
−2
=
= 0
(3)
∂x−2
Provide all the steps like this and values for other nodes in computation graph.
Figure 1: Two curves with equal area under the ROC curve
In the above figure we have ROC curves for two classifiers (A and B) which have equal areas under the curve (AUC).
-
Which classifier is better among these two? (write your detailed thinking)
-
Describe a situation in which undoubtedly classifier A should be preferred.
-
Describe a situation in which undoubtedly classifier B should be preferred.
-
Which factors determine the area under the curve?
∂∂ℓx,
Problem 6 (20 points)
Technically we need to compute the gradient with respect to Wi, the linear
transform (or parameter) matrix (tensor) for layer i. Yet, we are computing gradient of the loss ℓ with respect to the input x. How come?
Bonus for undergraduates beyond this line
Problem 7 (20 points)
Compare the following metrics and explain which one is better.
(a) ROC and AUC
Figure 2: Accuracy
-
For the same figure, explain relevance to True Positive Rate (TPR) and False Positive Rate (FPR)
4
