Quantcast
Channel: Active questions tagged python - Stack Overflow
Viewing all articles
Browse latest Browse all 23131

scrapy stops crawling with 500 Internal Server Error

$
0
0

I am crawling a web with scrapy and I receive the error:

Gave up retrying <GET https://www.something.net> (failed 3 times): 500 Internal Server Error

even though in the parse method I have added this parameter to the meta of the scrapy.Request that calls the parse function:

"handle_httpstatus_all": True,

Then in the parse function I do:

item = response.meta['item']if response.status == 200:    #Keeps building the itemyield item

So in theory this should not happen. What can I do to avoid it?


Viewing all articles
Browse latest Browse all 23131

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>