Comment on page

Start Simple

How to start simple with deep learning models?
Start Simple - Troubleshooting


  • Choose simple architecture:
    • LeNet/ResNet for images.
    • LSTM for sequences.
    • Fully-connected network with one hidden layer for all other tasks.
  • Use sensible hyper-parameter defaults:
    • Adam optimizer with a “magic” learning rate value of 3e-4.
    • ReLU activation for fully-connected and convolutional models and TanH activation for LSTM models.
    • He initialization for ReLU and Glorot initialization for TanH.
    • No regularization and data normalization.
  • Normalize data inputs: subtracting the mean and dividing by the variance.
  • Simplify the problem:
    • Working with a small training set around 10,000 examples.
    • Using a fixed number of objects, classes, input size, etc.
    • Creating a simpler synthetic training set like in research labs.
Last modified 3yr ago