Full Stack Deep Learning
Search…
Full Stack Deep Learning
Course Content
Setting up Machine Learning Projects
Infrastructure and Tooling
Data Management
Machine Learning Teams
Training and Debugging
Testing and Deployment
Research Areas
Labs
Where to go next
Guest Lectures
Xavier Amatriain (Curai)
Chip Huyen (Snorkel)
Lukas Biewald (Weights & Biases)
Jeremy Howard (Fast.ai)
Richard Socher (Salesforce)
Raquel Urtasun (Uber ATG)
Yangqing Jia (Alibaba)
Andrej Karpathy (Tesla)
Jai Ranganathan (KeepTruckin)
Franziska Bell (Toyota Research)
Corporate Training and Certification
Corporate Training
Certification
Powered By GitBook
Andrej Karpathy (Tesla)
Andrej is currently Senior Director of AI at Tesla,
 and was formerly a Research Scientist at OpenAI. His educational materials about deep learning remain among the most popular.
Programming The Software 2.0 Stack

Andrej's Landmark Post - "Software 2.0"

  • Software 1.0 consists of explicit instructions to the computer written by a programmer.
  • Software 2.0 can be written in much more abstract, human unfriendly language, such as the weights of a neural network.
  • In software 2.0, we restrict the search to a continuous subset of the program space where the search process can be made efficient with back-propagation and stochastic gradient descent.
Source: https://medium.com/@karpathy/software-2-0-a64152b37c35

Programming In The 2.0 Stack

  • If optimization is doing most of the coding, what are the humans doing?
    • 2.0 programmers label data
    • 1.0 programmers maintain the surrounding "dataset infrastructure":
      • Visualize data
      • Create and edit labels
      • Bubble up likely mislabeled examples
      • Suggest data to label
      • Flag labeler disagreements
  • Data labeling is highly iterative and non-trivial.
    • Lane lines are different across the world.
    • Cars have different shapes and sizes.
    • Even traffic lights and traffic signs can be ambiguous.
  • Label imbalances are very frequent.
  • Data imbalances are very common.
⇒ Realistic datasets: high label and data imbalances, noisy labels, highly multi-task, semi-supervised, active.

2.0 IDEs

  • Show a full inventory and statistics of the current dataset.
  • Create and edit annotation layers for any data point.
  • Flag, escalate, and resolve discrepancies in multiple labels.
  • Flag and escalate data points that are likely to be mislabeled.
  • Display predictions on an arbitrary set of test data points.
  • Auto-suggest data points that should be labeled.
⇒ Can we build GitHub for Software 2.0?
Guest Lectures - Previous
Yangqing Jia (Alibaba)
Next - Guest Lectures
Jai Ranganathan (KeepTruckin)
Last modified 2yr ago
Copy link
Outline
Andrej's Landmark Post - "Software 2.0"
Programming In The 2.0 Stack
2.0 IDEs