Full Stack Deep Learning
Full Stack Deep Learning
Setting up Machine Learning Projects
Infrastructure and Tooling
Machine Learning Teams
Training and Debugging
Testing and Deployment
Where to go next
Xavier Amatriain (Curai)
Chip Huyen (Snorkel)
Lukas Biewald (Weights & Biases)
Jeremy Howard (Fast.ai)
Richard Socher (Salesforce)
Raquel Urtasun (Uber ATG)
Yangqing Jia (Alibaba)
Andrej Karpathy (Tesla)
Jai Ranganathan (KeepTruckin)
Franziska Bell (Toyota Research)
Corporate Training and Certification
How to implement and debug deep learning models?
Debug - Troubleshooting
The 5 most common bugs in deep learning models include:
Incorrect shapes for tensors.
Pre-processing inputs incorrectly.
Incorrect input to the loss function.
Forgot to set up train mode for the network correctly.
Numerical instability - inf/NaN.
3 pieces of general advice for implementing models:
a lightweight implementation
such as Keras if possible, since most of the stuff in Keras works well out-of-the-box.
complicated data pipelines later
The first step is to
get the model to run
shape mismatch and casting issues
, you should step through your model creation and inference step-by-step in a debugger, checking for correct shapes and data types of your tensors.
, you can scale back your memory-intensive operations one-by-one.
, simply Google it. StackOverflow would be great most of the time.
The second step is to have the model
overfit a single batch
Error goes up:
Commonly this is due to a flip sign somewhere in the loss function/gradient.
This is usually a numerical issue, but can also be caused by a high learning rate.
You can lower the learning rate and inspect the data for shuffled labels or incorrect data augmentation.
You can increase the learning rate and get rid of regularization. Then you can inspect the loss function and the data pipeline for correctness.
The third step is to
compare the model to a known result
The most useful results come from
an official model implementation
evaluated on a similar dataset to yours
If you can’t find an official implementation on a similar dataset, you can compare your approach to results from
an official model implementation evaluated on a benchmark dataset
If there is no official implementation of your approach, you can compare it to results from
an unofficial model implementation
Then, you can compare to results from
a paper with no code
, results from
the model on a benchmark dataset
, and results from
a similar model on a similar dataset
An under-rated source of results come from
, which can help make sure that your model is learning anything at all.