Goal: Parameterize multiple paths and merge all data on ID using pandas. For example, I have multiple paths below:
- \Users\Documents\202301\data01.pqt
- \Users\Documents\202302\data02.pqt
- \Users\Documents\202303\data03.pqt...
- \Users\Documents{year}{month}\data{month}.pqt
I have no problems with parametrizing and merging the data but would like to know if there is a way to do a for loop on these files months/years at once and store/read/merge all parquet files within the loop.
I took a long route by storing and reading each file for once which takes a lot of manual work. Looking to automate this process and eliminate the need to create new code for each new file by using a for loop on the year and month within the file name.