Frequent Asked Question

Question1:
What is the process for updating jobs after they've been run by Airflow?

Answer: When the execution ends, the job will be update through the update_mongo function of the scheduler hook. If the execution ends sucessfully, the state of the job will be set to pending and the execution date will be incremented with n numer of days. Otherwise, when the execution fails, the job state will be updated to failed. The number of days to add for domain platforms are defined in the domain settings as fequencies. And fo te social media part it is set to 1 day for the stories and 7 days for the rest of tasks

Question2: How does the system handle rate limiting or API restrictions from the platforms being scraped ?

Answer: For the current version of the project does not hundel rate limitation for the drivers that uses API. When there is no quota or a error to lunch the crawl the jobs are considerate as failed and will be executed again until they are executed successfully.