Quantcast
Channel: Active questions tagged python - Stack Overflow
Viewing all articles
Browse latest Browse all 23160

Snowflake Python Connector - fetch_pandas_all() is really slow with less than 25k records

$
0
0

I'm using fetch_pandas_all function to retrieve result of a query that takes less than 2 sec in Snowflake (checked activity in UI) but the function itself takes like 30 seconds - we are talking about a query that returns about 25k records.

Versions that I use:

snowflake-connector-python 3.1.0pyarrow                    10.0.1pandas                     2.0.3

Code snippet:

    cursor = conn.cursor()    #cursor.arraysize = 50000    query = union_query    logging.info(f"Executing UNION query")    cursor.execute(query)    logging.info(f"Fetching UNION query result")    df = cursor.fetch_pandas_all()    cursor.close()

Setting arraysize has no effect on performance.

OUTPUT:

2023-08-22 14:07:43,633 | root | INFO | Successfully connected to Snowflake2023-08-22 14:07:43,633 | root | INFO | Executing UNION query2023-08-22 14:07:43,831 | root | INFO | Fetching UNION query result2023-08-22 14:08:16,246 | root | INFO | Finished fetching. Returned 26,625 rows

Viewing all articles
Browse latest Browse all 23160

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>