I am trying to fix this code to extract data from it. The code is publicly available here: https://github.com/grgmiller/CAISO_operations/blob/master/CAISO_operations.py
This code used to worked in 2020, but currently does not due to slight changes in the website, and its use of now-depreciated methods.
The first thing I did was to downgrade selenium to version 4.0.0 to factor out the errors created by the depreciated methods. Next, I added one line of code to the downloadCurtialment function after
curtail_read = pd.read_csv(curtailments / 'curtailment_data.csv', dtype=ct_dtypes) #load the csv into a dataframe
:
curtail_read = curtail_read.drop(columns=['Reason']) # Reason is not needed
The 'Reason' column was not originally in the dataset used in the code, so I dropped it from the dataframe.
The problem I am currently having is with the second to last line of code in the downloadDemand function:
browser.find_element_by_id('downloadNetDemandCSV').click() #download CSV file
I get this error output:
File "/home/luis/Research/CAISO_operations/CAISO_operations.py", line 101, in maindownloadDemand(browser, dataDate)File "/home/luis/Research/CAISO_operations/CAISO_operations.py", line 253, in downloadDemandbrowser.find_element_by_id('downloadNetDemandCSV').click() #download CSV fileFile "/home/luis/.local/lib/python3.10/site-packages/selenium/webdriver/remote/webelement.py", line 80, in clickself._execute(Command.CLICK_ELEMENT)File "/home/luis/.local/lib/python3.10/site-packages/selenium/webdriver/remote/webelement.py", line 693, in _executereturn self._parent.execute(command, params)File "/home/luis/.local/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 418, in executeself.error_handler.check_response(response)File "/home/luis/.local/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_responseraise exception_class(message, screen, stacktrace)selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable(Session info: chrome=123.0.6312.58)
I am a little bit lost about this error since it can do
browser.find_element_by_id('downloadDemandCSV').click() #download CSV file
just fine with downloadDemand.csv, and I checked the browser console, and both 'downloadDemandCSV'and 'downloadNetDemandCSV' are there in the dropdown medium. I cannot figure out what can be there error. Here is the website the code references. http://www.caiso.com/TodaysOutlook/Pages/default.aspx#section-net-demand-trend
Thank you all for the help, and apologize if it is something trivial I am new to selenium and web scraping in general.