Processing

What are efficient ways to process your data?
Processing - Data Management

Summary

  • The simplest thing we can do is a Makefile to specify what action(s) depend on.
  • You will probably need a workflow management system. Airflow is the current winner of this space.
  • Try to keep things simple and don't over-engineer your processing pipeline.