scrapy爬虫错误记录

相关包的版本

twisted.internet.error.TCPTimedOutError: TCP connection timed out: 10060: 由于连接方在一段时间后没有正确答复或连接的主机没有反应，连接尝试失败

原因：被检测到为爬虫
解决：爬虫时间设置为白天

[<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]
解决方案：setting设置useragent(https://github.com/scrapy/scrapy/issues/2513,https://stackoverflow.com/questions/40665120/under-what-circumstances-does-scrapy-throw-a-connection-was-closed-cleanly-err)

[<twisted.python.failure.Failure twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion: Connection lost.>]
解决方案https改为http,(https://newbedev.com/python-twisted-web-newclient-responseneverreceived-twisted-python-failure-failure-twisted-internet-error-connectionlost-connection-to-the-other-side-was-lost-in-a-no-n-clean-fashion-connection-lost-code-example)
使用虚拟环境让cryptography包版本<2
换代理（不是这个原因，https://support.zyte.com/support/discussions/topics/22000009971）

这篇好文章是转载于：学新通技术网