Description
-
Problem statement
The goal of the mini-projects is to implement a Noise2Noise model (Section 7.3 of the course). A Noise2Noise model is an image denoising network trained without a clean reference image. The original paper can be found at https://arxiv.org/abs/1803.04189.
The project has two parts, focusing on two di erent facets of deep learning. The rst one is to build a network that denoises using the PyTorch framework, in particular the torch.nn modules and autograd. The second one is to understand and build a framework, its constituent modules, that are the standard building blocks of deep networks without PyTorch’s autograd.
1.1 Submission instructions that apply to both projects
We will evaluate your codes with an automated testing suite, so your submission should adhere to the following structure.
Proj SCIPER1 SCIPER2 SCIPER3
Miniproject 1
init .py
model.py
bestmodel.pth
Report 1.pdf
others
otherfile1.py
otherfile2.py
Miniproject 2
init .py
model.py
bestmodel.pth
others
otherfile1.py
otherfile2.py
Report 2.pdf
Do not submit any IPython notebooks, as they will not be checked. others refers to all the codes that you want to include in your submission. You may choose to structure those di erently. The immutable ones are the two folders named Miniproject 1, Miniproject 2, model.py in both those folders with the speci c structure described above.
A zip le named Proj SCIPER1 SCIPER2 SCIPER3.zip must be uploaded to the Moodle of the course before Friday May 27th, 2022 23:59PM . This zip le should uncompress into two directories named Miniproject_1 and Miniproject_2, with the directory structure given above.
When needed, you should add comments to your source codes to facilitate their understanding.
The code must work in the VM provided for the course, in particular, it should not require additional software or libraries.
Exchange of code or report snippets between groups is forbidden. Using code or really anything that is not your original creation without citing it is called plagairism { it is wrong and will be treated very seriously. Every student should have a clear understanding of their group’s entire source code and report. This will be checked during the oral presentation.
-
Mini-project 1: Using the standard PyTorch framework
You have, at your disposal, the whole PyTorch framework, with all its powerful modules to implement the denoiser. You are free to explore the various modules available and build any network that you deem necessary. It should be implemented with PyTorch only, in particular, without using other external libraries such as scikit-learn or NumPy.
You should explore various parts of the training pipeline like data augmentation strategies, optimization methods, loss functions, etc.
Training
You have been provided a train data.pkl le with two tensors of the size 50000 3 H W . You can load this le with a command like:
-
import torch
-
noisy_imgs_1 , n o i s y _ i m g s _ 2 = torch . load ( ’t r a i n _ d a t a . pkl ’)
This constitutes the training data and corresponds to 50000 noisy pairs of images. Each of the 50000 pairs provided correspond to downsampled, pixelated images. Train a network that uses these two tensors to denoise i.e., reduce the e ects of downsampling on unseen images.
An additional val data.pkl is provided so that you can track your progress. You can load the validation le with a command like:
-
noisy_imgs , c l e a n _ i m g s = torch . load ( ’val_data . pkl ’)
Your proposed method and network will be evaluated on a di erent set of images than the ones given here. They will also be of size 3 H W , but their noise characteristics may vary.
Evaluation and Submission
The nal version of your code should contain a model.py that will be imported by the testing pipeline.
This le should contain a class :
1 ### For mini – project 1
-
class Model () :
-
def __init__ ( self ) -> None :
|
4 |
## i n s t a n t i a t e model + o pt im iz er + loss function + any other stuff you need |
|
5 |
pass |
|
6 |
|
|
7 |
def l o a d _ p r e t r a i n e d _ m o d e l ( self ) -> None : |
|
8 |
## This loads the p a r a m e t e r s saved in b es tm od el . pth into the model |
|
9 |
pass |
|
10 |
-
def train ( self , train_input , train_target , n u m _ e p o c h s ) -> None :
|
12 |
#: t r a i n _ i n p u t : tensor |
of |
size (N, C, H, W) c o n t a i n i n g a noisy version of the images |
|
. |
|||
|
13 |
#: t r a i n _ t a r g e t : tensor |
of |
size (N, C, H, W) c o n t a i n i n g another noisy version of the |
|
same images , which only differs from the input by their noise . |
|||
|
14 |
pass |
||
|
15 |
|||
|
16 |
def predict ( self , t e s t _ i n p u t ) |
-> torch . Tensor : |
|
|
17 |
#: t e s t _ i n p u t : tensor |
of size (N1 |
, C, H, W) with values in |
range 0 -255 that has to |
|||
|
be |
denoised by the trained or |
the |
loaded |
network . |
|||
|
18 |
#: returns a tensor of |
the |
size (N1 , C, H, |
W) with values |
in range 0 -255. |
||
|
19 |
pass |
||||||
The Model class will serve as the entry-point to your code for evaluation. Its constructor should not take any inputs. The required interface of the Model is shown with comments about what each required function should do. In the Model class, the predict method takes as input a Tensor with values in range 0-255 and should return a Tensor with values in range 0-255.
Using this interface, a model will be trained, and will be evaluated on a test set. Additionally, you should provide your \best” model that you think will perform the best on an unseen test set. You should save this model under \bestmodel.pth” and the model class should provide a function load pretrained model that loads this model. For that purpose, you can have a look at model serialization methods for PyTorch at https://pytorch.org/ docs/stable/notes/serialization.html#saving-and-loading-torch-nn-modules.
Ensure that your code can runs on GPUs, and on CPU only systems, without any changes to the code. The easiest way to make this happen is to de ne a device as
-
device = torch . device ( ” cuda “ if torch . cuda . i s _ a v a i l a b l e () else ” cpu” )
and use this to de ne the storage location of each of your tensors. For example,
-
all_ones = torch . ones (2 , 3) . to ( device )
will use GPU capabilities if PyTorch detects one, else seamlessly works on the CPU.
Evaluation of the submitted models will be based on the Peak Signal-to-Noise Ratio (PSNR) metric. An extremely simpli ed implementation of PSNR for a single input image, with pixel values in [0; 1], looks like
-
def psnr ( denoised , g r o u n d _ t r u t h ) :
|
2 |
# Peak |
Signal to |
Noise Ratio : denoised and |
g r o u n d _ t r u t h have range [0 , 1] |
|
3 |
mse = |
torch . mean |
(( denoised – g r o u n d _ t r u t h ) |
** 2) |
-
return -10 * torch . log10 ( mse + 10** -8)
A network with 8 convolutional layers achieves 24 dB PSNR on the validation data provided (the function above returns PSNR in dB). An extremely simple benchmark achieves 23 dB, so if you are not reaching this you will want to assume that there is a problem with your implementation.
A suggestion: You can consider experimenting with a subset of the training data, and train on the full dataset after you’ve made your architectural, training choices etc. You can choose services like Google Colab for larger computation power.
We provide a test.py that uses a similar interface to the one that we will use for the grading, which trains your model and then evaluates it on a validation set. The training and testing should take at most 10 min on a computer with a small GPU (at most 45 min on a CPU). If your code does not follow the interface or crashes during evaluation, you will not get any points for the code submission.
