How to implement and debug deep learning models?
Debug - Troubleshooting


  • The 5 most common bugs in deep learning models include:
    • Incorrect shapes for tensors.
    • Pre-processing inputs incorrectly.
    • Incorrect input to the loss function.
    • Forgot to set up train mode for the network correctly.
    • Numerical instability - inf/NaN.
  • 3 pieces of general advice for implementing models:
    • Start with a lightweight implementation.
    • Use off-the-shelf components such as Keras if possible, since most of the stuff in Keras works well out-of-the-box.
    • Build complicated data pipelines later.
  • The first step is to get the model to run:
    • For shape mismatch and casting issues, you should step through your model creation and inference step-by-step in a debugger, checking for correct shapes and data types of your tensors.
    • For out-of-memory issues, you can scale back your memory-intensive operations one-by-one.
    • For other issues, simply Google it. StackOverflow would be great most of the time.
  • The second step is to have the model overfit a single batch:
    • Error goes up: Commonly this is due to a flip sign somewhere in the loss function/gradient.
    • Error explodes: This is usually a numerical issue, but can also be caused by a high learning rate.
    • Error oscillates: You can lower the learning rate and inspect the data for shuffled labels or incorrect data augmentation.
    • Error plateaus: You can increase the learning rate and get rid of regularization. Then you can inspect the loss function and the data pipeline for correctness.
  • The third step is to compare the model to a known result:
    • The most useful results come from an official model implementation evaluated on a similar dataset to yours.
    • If you can’t find an official implementation on a similar dataset, you can compare your approach to results from an official model implementation evaluated on a benchmark dataset.
    • If there is no official implementation of your approach, you can compare it to results from an unofficial model implementation.
    • Then, you can compare to results from a paper with no code, results from the model on a benchmark dataset, and results from a similar model on a similar dataset.
    • An under-rated source of results come from simple baselines, which can help make sure that your model is learning anything at all.