What are efficient ways to process your data?


  • The simplest thing we can do is a Makefile to specify what action(s) depend on.

  • You will probably need a workflow management system. Airflow is the current winner of this space.

  • Try to keep things simple and don't over-engineer your processing pipeline.

Last updated