Skip to main content
774 questions
Newest Active Bountied Unanswered
0 votes
0 answers
14 views

The inciting problem One of our DAGs suffers from an issue wherein a BigQueryInsertJobOperator task fails due to it becoming a "zombie task" (from Airflow's perspective); in the "Event ...
IgorM's user avatar
  • 198
3 votes
2 answers
179 views

Let's say I have a DAG with some source1, source2, ... tasks and a sink task, with a simple [source1, source2, ...] >> sink dependency. All of them are @task-annotated Python operators, and the ...
Vito De Tullio's user avatar
  • 1,245
Advice
0 votes
0 replies
55 views

I use docker-compose to run airflow. Postgres section is postgres: image: postgres:12.16 environment: - POSTGRES_USER - POSTGRES_PASSWORD - POSTGRES_DB healthcheck: test: [ "...
Cherry's user avatar
  • 34k
Best practices
0 votes
0 replies
49 views

Are there any best practices for handling roles in code? Most of our setup is managed via Terraform. It would be great if there was a possibility. There is a Terraform provider, however using the ...
Moritz Schmitz v. Hülst's user avatar
  • 3,761
Tooling
1 vote
0 replies
74 views

I’m using Apache Airflow 2.4.3 and trying to securely store a Snowflake connection with a private key inside the connection’s extras JSON field. I want to mask the sensitive private_key_content field ...
LIONEL JOSEPH's user avatar
  • 43
0 votes
0 answers
82 views

I have added a airflow_local_setting.py file to my mwaa s3 bucket at the root of dags folder. I have a dag_policy defined which is being picked up and works as expected. In the same file I have added ...
hello's user avatar
  • 13
0 votes
0 answers
76 views

I am using Google Cloud Composer environment and Apache Airflow to run my DAG. Almost all tasks are run in deferrable mode, since they are long-running. I noticed that when running a backfill job for ...
Murphy's user avatar
  • 1
0 votes
1 answer
59 views

I have a task dependency like this: task1 (must succeed) | v +-------------------+ | | task2-A task2-B | | v ...
notmegg's user avatar
  • 1
0 votes
0 answers
44 views

We're migrating some scheduled jobs from to Airflow. The DAGs are written at some point in time, tested and deployed to the production environment. Another team is responsible for enabling these DAGs ...
Greg B's user avatar
  • 15k
0 votes
0 answers
39 views

I am using Airflow with the Bitnami Helm chart (version 17.2.4). I added a custom JSON logger using a ConfigMap and set the environment variable AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=...
Sahil Gupta's user avatar
  • 1
0 votes
0 answers
27 views

How can I pass the task_id parameter through curl? I use KubernetesPodOperator After naming the variable tasks = """ {{ dag_run.conf.image_tag }} """ and specifying ...
nxnos's user avatar
  • 1
0 votes
0 answers
29 views

I want to create a logic that can run 2 DAGs in parallel. When more DAGs triggered and 2 DAGs already running I want to skip the upcoming DAGs. Current logic queue the upcoming DAGs and not skip them ...
Solomon Shubaev's user avatar
  • 33
0 votes
0 answers
215 views

I updated my MWAA environment from 2.4.3 to 2.10.3 and I keep getting the following error randomly when i am executing multiple task at the same time: INFO - Task exited with return code -9. For more ...
Justin's user avatar
  • 121
0 votes
1 answer
271 views

I’m using Apache Airflow with the LocalExecutor on a machine that has an 8-core CPU. Let’s say I have a DAG with two tasks that can run in parallel, and each of these tasks uses Python's ...
Aaditya 's user avatar
  • 23
2 votes
1 answer
296 views

Below is a minimal implementation of a branch operator using the taskflow api. The dag will execute either odd_task or even_task based on the string given by branch_on_condition. odd_task or even_task ...
GazYah's user avatar
  • 38

15 30 50 per page
1
2 3 4 5
…
52