I'm facing an issue with the loading time of data from PostgreSQL to python Polars. I have 7 tables. two tables have size of 1.5 gb and others around 500 mb. I load the table into various polars data frames and the joined based on single column. it takes around 20 minutes to load all table and joins to make a single data frame. Also the memory is so high when joining. Is there any solution which can reduce the all the time to 10 -30 seconds. Also lower time is appreciated. any tools or any postgresql tricks
n_live_tup is rows count.
-[ RECORD 1 ]------------------
schemaname | public
relname | table1
n_live_tup | 7467575
-[ RECORD 2 ]------------------
schemaname | public
relname | table2
n_live_tup | 1341294
-[ RECORD 3 ]------------------
schemaname | public
relname | table3
n_live_tup | 888511
-[ RECORD 4]------------------
schemaname | public
relname | table4
n_live_tup | 112289
-[ RECORD 5 ]------------------
schemaname | public
relname | table5
n_live_tup | 47007
-[ RECORD 6 ]-----------------
schemaname | public
relname | table6
n_live_tup | 12722