I'm using fetch_pandas_all function to retrieve result of a query that takes less than 2 sec in Snowflake (checked activity in UI) but the function itself takes like 30 seconds - we are talking about a query that returns about 25k records.
Versions that I use:
snowflake-connector-python 3.1.0pyarrow 10.0.1pandas 2.0.3Code snippet:
cursor = conn.cursor() #cursor.arraysize = 50000 query = union_query logging.info(f"Executing UNION query") cursor.execute(query) logging.info(f"Fetching UNION query result") df = cursor.fetch_pandas_all() cursor.close()Setting arraysize has no effect on performance.
OUTPUT:
2023-08-22 14:07:43,633 | root | INFO | Successfully connected to Snowflake2023-08-22 14:07:43,633 | root | INFO | Executing UNION query2023-08-22 14:07:43,831 | root | INFO | Fetching UNION query result2023-08-22 14:08:16,246 | root | INFO | Finished fetching. Returned 26,625 rows