Summary of Krizhevsky et. al.‘s 2012 paper
Introduction:
AlexNet is a deep Convolutional Neural Network based on LeNet by LeCun, but, of course, a much much improved network.
this model by Alex Krizhevski is the winner of the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) in 2012 and it made a big impact as beginning from that year, teams from leading universities, companies and startups have competed every year in ILSVRC to claim the State-of-the-art performance of the ImageNet dataset.
However, its purpose, as any machine learning algorithm, is to reduce over-fitting, provide optimized and optimizing implementations to make training faster and to reduce error rate of course.
Procedures:
Pre-processing of the images
AlexNet system requires a constant input dimensionality of 256 x 256, so they simply re-scaled their input images.
Architecture
the network has 60 million parameters and 650 000 neurons. It consists of 5 convolutional layers, some of which are followed by max-pooling layers and 3 fully connected layers with a final 1000-way softmax activation function.
Reducing over-fitting
Alex used data augmentation which I guess the oldest trick in the industry for this purpose ,maybe? well it’s efficient.
Also they used Dropout in the first two fully-connected layers which they found is very useful saying that AlexNet “exhibits substantial over-fitting”.
Results:
Krizhevski’s Net achieved top-1 and top-5 test error rates of 37.5% and 17.0% , comparing it to the best published results in that time, 45.7% and 25.7% respectively, that’s quite an improvement.
Conclusion:
The results of this network are quite amazing seeing how building a deep CNN can provide impressing outputs and error rates comparing to older systems
“for example, removing any of the middle layers results in a losing of about 2% for the top-1 performance of the network. So the depth really is important for achieving our results”
Personal notes:
Whenever I read an article about deep convolution network, 80% of the time, it mentioned the AlexNet in it, so, pretty much, the improvement it presents are considered to be groundbreaking achievement and merits the State-of-the-art title for that.
