Machine Learning Assignment #3

Description

1. Password protect your ZIP file using a password with 8-10 characters. Use only alphanu-meric characters (a-z A-Z 0-9) in your password. Do not use special characters, punctu-ation marks, whitespaces etc in your password. Name your ZIP filesubmit.zip. Specify the file name properly in the Google form.

else’s code/algorithm but there would be heavy penalty for doing so without giving proper credit to that person/source.

2. Train your chosen method on the train data we have provided (using any validation technique you feel is good). You may or may not wish to use the reference images

– it is your wish. Store the model you obtained in the form of any number of bi-nary/text/pickled/compressed files as is convenient and write a prediction method in Python in the file predict.py that can take new images and use your model files you have stored to make predictions on that image. Include all the model files, thepredict.py file, as well as any library files that may be required to run the prediction code, in your ZIP submission. You are allowed to have subdirectories within the archive.

(40 + 60 marks)

Evaluation Measures and Marking Scheme . Part 1 needs to be answered in the PDF file itself and part 2 needs to be answered in the ZIP file submission. We will evaluate your method on our secret test data which will have 500 × 150 images, each having 3 Greek upper case characters with similar kinds of rotations and background obfuscations as the training images. The 60 marks for part 2 will be awarded as follows:

1. Total size of the submission, once unzipped (smaller is better): 10 marks

2. Average time taken to process a test image (smaller is better): 10 marks

3. Code match (procedure given below): 40 marks

Marks for parts 1 and 2 would be awarded in a relative manner after it is known how all groups have performed in terms of model size and prediction time. However, marks for parts 3 and 4 would be awarded in an absolute sense, as outlined below:

1. Your method must output, for a test image the names of the 3 Greek characters in that image. The names must be spelt out in upper case Latin (please see the labels.txt and images in the reference folder to see how to spell the Greek letter names correctly). Names must be separated by a single comma with no trailing/stray whitespaces or other punctuation marks.

2. If the code returned by your routine for an image contains more than 3 names, we will only take the first 3 names and discard the rest (thus, we will truncate your code at 3 characters).

3. We will then count how many names were in their correct locations. Suppose ^ˆl names were in correct locations and spelt correctly then you will receive a (fractional) score of ₃^ˆl for this image.

4. Make sure there are no spelling mistakes while writing the names of the characters – see labels.txt and images in the reference folder to see how to spell the Greek letter names correctly.

5. We will sum these scores over all test images, divide this sum by the number of test images and multiply that by 40 to award you marks out of 40 for part 3.

6. Please see eval.py supplied with this package (in the folder sample_submit) to see how this evaluation script will work. We will use a similar script to actually evaluate your submissions.

7. Please make sure your code does run with eval.py. Try running your evaluation model and predict.py file with a small set of 100-200 images from the train set to check for coding glitches or other errors.

Suggested Methods. Various techniques may be used to successfully solve this problem. Take care to use powerful and nonlinear techniques e.g. kernels, deep learning as the final step. Expecting a deep network to take the whole image and solve the problem in one shot may be tempting, but also unnecessary and may end up giving larger model size and prediction times. We leave the final decision on the method to you, but offer some useful steps below.

1. Identify background pixels:

1. 1. Notice that background pixels are always a light shade whereas the obfuscating lines are always a darker shade. Can you learn from the training set what is the distribu-tion of shade/color of the background pixels?

1. 2. To do so first you will have to extract the background color of an image so that you can perform estimation. However, this should be easy since the corners of the image are almost always background.

1. 3. You may want to represent pixel color in the HSV/HSL format instead of the more popular RGB format to easily identify shade.

2. Identify pixels that belong to obfuscating lines:

1. 1. Can you similarly learn a distribution on the shade/color of the obfuscating lines?

1. 2. Another technique that may help eliminate obfuscating lines is a step called erosion. Check this step out in the opencv library. This step will thin out lines so that the obfuscating lines may get very thin or even disappear. You will have to learn a good value of erosion parameter.

1. 3. Using any method (including but not limited to the above), once you have found a way to identify which pixels in an image belong to obfuscating lines, use your method(s) to eliminate obfuscating lines in images.

3. Segment image into pieces, each containing one char:

1. 1. You may want to perform segmentation (a form of pixel clustering in images to club together adjacent pixels that are similar) to find out how many characters are present in an image. There are several segmentation techniques available (several are preimplemented) in the opencv library.

1. 2. Another technique that may help you segment the image into pieces is by looking for vertical columns of pixels that contain very few non-background pixels. This should be easy once we have learnt to identify background pixels in the previous stage. The reason this helps is that regions where there is a char, a lot of the pixels are non-background which makes it clear that a char is present.

4. Use ML models to identify the char in each piece:

1. 1. You may want to remove color information from the image at this stage since colors cannot help you identify which char is present in a piece (or can they?). Also, try to trim the piece so that it contains mostly the char and not too much blank space.

1. 2. The reference images may help here since the rotations which have been applied to the chars are not arbitrary but small multiples of 10 degrees.

3. To exploit the above, why not create rotated versions of the reference images and see which one most closely matches the char within a given piece?

3. An alternative is to instead attempt to rotate back the piece itself so that it matches one of the standard reference images.

3. Try simple (linear and/or non-linear) techniques to classify every piece into one of the 24 characters. May treat this as a multiclass problem.

Resources The OpenCV library https://opencv.org/ provides routines for most operations you may need to solve this problem. OpenCV is well integrated into Python and available via pip. More challenging versions of this DeCAPTCHA problem are problems of identifying license number plates on cars (e.g. https://sod.pixlab.io/articles/license-plate-detection. html) and identifying house numbers from images of doors (e.g. https://www.kaggle.com/ stanfordu/street-view-house-numbers) etc. You may search for approaches people have used to solve those problems to get motivation. However, be warned that the solutions to those problems may be an overkill for our much simpler problem and blindly using those techniques may result in large model sizes or prediction times, both of which would cause you to lose marks.

Code Usage from the Internet. You have full freedom to download code available online (written by others) for training without any penalty (even if you are able to directly use some-one’s code without making any changes to it). However you must cite all code that you take from others by giving names of authors and name of the URLs from where you obtained said code. There is no penalty for using code for which you have cited the original author. However, there will be heavy penalty for using someone’s work and not giving them credit.

• How to Prepare the PDF File

Use the following style file to prepare your report. https://media.neurips.cc/Conferences/NeurIPS2022/Styles/neurips_2022.sty

For an example file and instructions, please refer to the following files https://media.neurips.cc/Conferences/NeurIPS2022/Styles/neurips_2022.tex https://media.neurips.cc/Conferences/NeurIPS2022/Styles/neurips_2022.pdf

Use the command \usepackage[preprint]{neurips_2022} in the preamble instead of \usepackage[final]{neurips_2022}. Use proper L^AT_EX commands to neatly typeset your responses to the various parts of the problem. Use neat math expressions to typeset any math-ematical derivations. All plots must be generated electronically – no hand-drawn plots would be accepted. All plots must have axes titles and a legend indicating what the plotted quantities are. Insert the plot into the PDF file using proper ^ALT_EX \includegraphics commands.

• How to Prepare the ZIP File

Your submission ZIP archive must contain a file called predict.py which must contain a method by the name decaptcha which we will call to decode the CAPTCHA images. We will pass a list of strings to the method decaptcha, each string corresponding to the path to a file name. The method must return a numpy array of numbers (containing predictions for the number of characters in each image) as well as a list of strings (containing the predictions for codes present in those images). The assignment package contains a sample submission filesample_submit.zip

which shows you how you must structure your submission as well as showing you the way in which your prediction code predict.py must accept input and return decodings. Do not change this input output behavior otherwise our autograder will not be able to process your recommendations and award you zero marks.

1. Your ZIP archive itself must contain a python file namedpredict.py which should not be contained inside any other directory within the archive. Apart from this file your ZIP archive may contain any number of files or even directories and subdirectories.

2. Do not change the name predict.py to anything else. The file must contain a method called decaptcha which must take in a list of filepaths to images and return a list of strings which should correspond to the labels for those images. Each label should be a list of 3 names of Greek alphabets, separated by a single comma with no whitespaces or other punctuation marks.

3. There are no “non-editable regions” in any of the files for this assignment. So long as you preserve the input output behavior, feel free to change the structure of the file as you wish. However, do not change the name of the filepredict.py to anything else.

4. Code you download from the internet may be in C/C++. You are free to perform training in C/C++ etc. However, your prediction code must be a python file calledpredict.py. This file may internally call C/C++ code but it is your job to manage that using extension modules etc. Do not expect us to call anything other than the single filepredict.py

5. The assignment package also contains a file called eval.py which is an example of the kind of file we will be using to evaluate the code that you submit. Before submitting your code, make sure to run eval.py and confirm that there are no errors etc.

6. Do not submit eval.py to us – we already have it with us. We have given you access to the fileeval.py just to show you how we would be evaluating your code.

7. Make sure that running predict.py does not require us to install any Python libraries which are not available using pip. Libraries that are installable using pip are fine. How-ever, if your prediction routine requires using external code not available using pip, you must supply all that external code to us within your ZIP archive – we should not have to install it. To be sure, we will not install or download any libraries from places like GitHub, personal homepages, or other repositories (even the repository mentioned in the question itself) even if you want us to. Your archive itself should contain all such files.

8. Other than the above restrictions, you are free to import libraries as you wish,e.g. pil, sklearn, scipy, pandas. Files for libraries not available on pip must be provided by you.

9. Once we have installed pip libraries required by your code, we should just be able to call predict.py and get predictions i.e. your prediction code must be self contained. We should not have to create any new directories, move files around or rename them. All of that must be done by you while creating the archive (remember, you are allowed to have as many files and (sub)directories apart from the filepredict.py as you want).

10. We do not need your training code. We simply need your prediction code. Do not increase your submission file size (remember there are marks for submission size too) by including unnecessary files.

11. The system on which we will run your prediction code will not have a GPU. Make sure your code can run simply on a CPU. Thus, use the CPU version of deep learning libraries if you are using them and not the GPU version of those libraries e.g. keras.

11. You may use a GPU (if you can find one) to train your method but your prediction method must still work on a CPU. To be sure, we will not offer your prediction code a GPU. Do not use any libraries that require GPU access during prediction. There are no restrictions on using GPU for training but we will not provide your code GPU access during prediction.

11. We will test your submitted code on a secret dataset which would have images of the same size as the training images. All test images will contain 3 upper case Greek characters.

9