I am crawling a web with scrapy and I receive the error:
Gave up retrying <GET https://www.something.net> (failed 3 times): 500 Internal Server Erroreven though in the parse method I have added this parameter to the meta of the scrapy.Request that calls the parse function:
"handle_httpstatus_all": True,Then in the parse function I do:
item = response.meta['item']if response.status == 200: #Keeps building the itemyield itemSo in theory this should not happen. What can I do to avoid it?