I have a question about for what to use airflow for with our current system.
I have an api which, by using GCP Cloud Tasks, trigger some tasks like sending a welcome email after 2 days and sending useful notifications. This is all in our main api repo.
Now I want for example add 1) a profile image face detector and notify users about the use of a non facial profile image. 2) multiple marketing reminder emails etc 3) classification of user profiles based on their data and 4) a lot more of these kind of background tasks as we grow
It feels weird to put all this extra logic and tasks in the main api repo, trigger it with cloud tasks/cloud scheduler etc is not exactly a good idea. Especially since some tasks can be reused/linked together. Also Cloud Tasks are just http calls but scheduled/delayed and the monitoring is not really intuitive.
So would Airflow be a good solution to manage these tasks and be future proof as our system/business needs besides the api grow?