What are efficient ways to process your data?
Last updated 4 years ago
The simplest thing we can do is a Makefile to specify what action(s) depend on.
You will probably need a workflow management system. Airflow is the current winner of this space.
Try to keep things simple and don't over-engineer your processing pipeline.
