Monitoring

How to monitor your machine learning system?
Monitoring - Testing and Deployment

Summary

  • It is crucial to monitor serving systems, training pipelines, and input data. A typical monitoring system can raise alarms when things go wrong and provide the records for tuning things.
  • Cloud providers have decent monitoring solutions.
  • Anything that can be logged can be monitored: dependency changes, distribution shift in data, model instabilities, etc.
  • Data distribution monitoring is an underserved need!
  • It is important to monitor the business uses of the model, not just its statistics. Furthermore, it is important to be able to contribute failures back to the dataset.