The goal of this project is to design a convolutional neural network to classify traffic signs. Other architectures are explored such as GoogLeNet. I started with the LeNet architecture and then added inception modules and dropout.

The goals/steps of this project are the following:

  • Load the data set (Download Here)
  • Explore, summarize and visualize the data samples
  • Design, train and test the model architecture
  • Use the model to make predictions on new images
  • Analyze the softmax probabilities of the new images

Training, Validation and Testing Data Size

I used numpy to calculate summary statistics of the traffic signs data set:

  • Size of training set: 34799
  • Size of the validation set: 4410
  • Size of test set: 12630
  • Shape of a traffic sign image: 32x32 (RGB)
  • The number of unique classes/labels: 43

Distribution of Training Data

Here is an exploratory visualization of the data set. The bar chart shows the distribution of training samples by class label.

Training data set distribution across the class labels

Convolutional Neural Network (CNN) Model Architecture

Preprocessing Data Sets

The first step is grayscaling the images to increase performance during training.
However, there may be some cases where the color can be significant, although I
haven't explored this.

The image is also normalized to improve training performance, making the grayscale pixel numerical values between 0.1-0.9 instead of the range from 0-255.

Here is an example of a traffic sign image before and after grayscaling and normalization.

Sample image before grayscale and after.

Model Architecture

My final model consisted of the following layers. I started with the LeNet architecture and then followed the paper on GoogLeNet by adding inception modules, average pooling followed by 2 fully connected layers with dropout in between both.

Layer Description
Input 32x32x3 RGB image
Preprocessing 32x32x1 Grayscale normalized image
Convolution 5x5 1x1 stride, valid padding, outputs 28x28x6
RELU 1 Activation
Convolution 5x5 1x1 stride, valid padding, outputs 24x24x16
RELU 2 Activation
Average pooling 1 2x2 stride, outputs 12x12x16
Inception Module 1 outputs 12x12x256
Average pooling 2 2x2 stride, outputs 6x6x256
Inception Module 2 outputs 6x6x512
Average pooling 3 2x2 stride, same padding, outputs 6x6x512
Dropout 1 Flatten the output and dropout 50% layer
Fully connected 1 Outputs 9216x512
Dropout 2 Dropout 50% layer
Fully connected 2 Outputs 512x43
Softmax Softmax probabilities

Training the Model

  • Preprocessed the images
  • One hot encoded the labels
  • Trained using Adam optimizer
  • Batch size: 128 images
  • Learning rate: 0.001
  • Variables initialize with truncated normal distributions
  • Hyperparameters, mu = 0, sigma = 0.1, dropout = 0.5
  • Trained for 20 epochs

Training was carried out on an AWS g2.2xlarge GPU instance. Each epoch was completed in around 24s, total training time of about 6-8 minutes.

Designing the Model

Final model results:

  • Validation set accuracy: 0.97
  • Test set accuracy: 0.95

The approach started with a review of LeNet and GoogLeNet architectures and those were used as a starting point to decide how to order and how many inception modules, pooling convolution and dropout layers to use.

  • First started with LeNet and with minor tweaks, achieved an accuracy of about 0.89
  • Reduced and switched max pooling to average pooling to get to 0.90
  • Added an inception module after the first two convolutions to get to 0.90-0.93
  • At this point, there were very deep inception modules (referencing GoogLeNet) and 4 fully connected layers and the model appeared to be overfitting.
  • Reducing the number of fully connected layers to 2 and adding dropout between them made the largest impact and achieved around 0.97 accuracy.

Once I noticed the accuracy starting high and surpassing the minimum 0.93, I tweaked the layers by trial and error but my approach was to try and keep as much information flowing through the model while balancing the dimensional reduction. This led me to have 2 convolution layers with small patch sizes, average pooling with small patch size and do more severe reductions with the inception modules.

Test a Model on New Images

Here are some other German traffic signs that I found on the web. The signs are
generally easy to identify, I chose a few that are at angles and one that had a
"1" spray painted onto a 30km/h sign.

Sample of test images that have not been seen by the network before

New Image Test Results

Here are the results of the predictions on the new traffic signs:

Image Prediction
Yield Yield
Traffic Signals Traffic Signals
General Caution General Caution
Speed Limit 30 Speed Limit 30
Right of Way Right of Way
Road Work Road Work
Bumpy Road Bicycles Crossing
No Entry No Entry
Speed Limit 100 Roundabout Mandatory
Speed Limit 30 (spray painted) Speed Limit 30

The model was able to correctly guess 8 of the 10 traffic signs, which gives an accuracy of 80%. I was impressed that the spray-painted sign still classified correctly even though the clear 100km/h sign was wrong.

The bicycles crossing and bumpy road signs do look very similar at low resolution, so I can understand that wrong prediction.

Model Certainty

The following shows the top 5 softmax probabilities for each sign.

Top 5 softmax probabilities for each sign.

Visualizing the Model Layers

It's interesting to see the layers of the network as it progresses. The convolution layers look the most interesting, while the average pooling layers start to look overly simplified. However, the average pooling images only show one layer in a very deep set.

Convolution 1
Convolution 2
Average Pooling 1
Average Pooling 2
Average Pooling 3

Conclusions and Discussion

Having LeNet and GoogLeNet as starting points was very helpful. There are still many areas for improvement by tweaking variables, moving/adding/removing layers and augmenting the data set. I noticed that when applying the GoogLeNet architecture directly to this data set, the training time was very high and accuracy plateaued quickly which could indicate over/underfitting. But in this case, I believe there were too many parameters, I found better performance by reducing the depth of the inception layers and adding dropout.

Check out the GitHub repo for the code and Jupyter Notebook.