Quantcast
Channel: Active questions tagged python - Stack Overflow
Viewing all articles
Browse latest Browse all 23305

Problem importing scripts in Apache Airflow - Modules Management

$
0
0

When I trying to import a script located in a folder outside dags I got a

ModuleNotFoundError: No module named 'scripts.send_email'

or

ImportError: cannot import name 'send_email' from 'scripts' (unknownlocation)

I've tried the following ways:

from airflow.scripts.send_email import send_email #ModuleNotFoundErrorfrom scripts.send_email import send_email #ModuleNotFoundErrorfrom scripts import send_email # ImportError

My Airflow Project has the following structure

airflow├── config├── dags├── __init__.py├── coinbase_fetch.py├── coinbase_orchestration.py└── load_into_minio.py ├── logs├── plugins├── scripts├── __init__.py└── send_email.py├── __init__.py├── docker-compose.py├── Dockerfile└── requirements.txt

The script I try to load in scripts/send_email.py is only an example which return a string. inside send email you'll find:

def send_email():   print("SEND EMAIL") 

I've added to the dockerfile the PYTHONPATH as I read here Modules Management. Therefore my Dockerfile is as you can see here:

FROM apache/airflow:2.9.0ENV PYTHONPATH "${PYTHONPATH}:/usr/local/airflow/scripts"ENV AIRFLOW__CORE__LOAD_EXAMPLES=TrueENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=my_conn_stringCOPY requirements.txt /RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt

I've also added the scripts file in the docker-compose volumes

  volumes:    - ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config    - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags    - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs    - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins    - ${AIRFLOW_PROJ_DIR:-.}/scripts:/opt/airflow/scripts #Added 

However I couldn't import the module inside the script folder.

I've also tried using sys.path.insert or sys.path.append as was mentioned in the links below without success:

The full project is in this github in the folder airflow: WIP Airflow


Viewing all articles
Browse latest Browse all 23305

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>