Scrapy -- 信号的使用

是大嘟嘟呀

于 2021-04-26 20:19:40 发布

阅读量2k

点赞数 1

分类专栏：网络爬虫 Scrapy 文章标签： python 爬虫开发语言

本文链接：https://blog.csdn.net/MarkAdc/article/details/116171126

版权

网络爬虫同时被 2 个专栏收录

32 篇文章

订阅专栏

Scrapy

6 篇文章

订阅专栏

代码

在爬虫文件中

	@classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super(T1Spider, cls).from_crawler(crawler, *args, **kwargs)
        spider.id = f"t1-{datetime.datetime.now()}"
        crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)
        crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
        crawler.signals.connect(spider.spider_opened, signal=signals.spider_opened)
        return spider

    def item_scraped(self, item, response, spider):
        if item:
            print(spider.id, "item_scraped")
            self.item_count += 1
            print("已采集 {} 个item".format(self.item_count))

    def spider_opened(self, spider):
        print("{}: 打开了".format(spider.id))

    def spider_closed(self, spider, reason):
        print("spider.id: {}".format(spider.id))
        print("reason: {}".format(reason))
        print("一共采集 {} 个item".format(self.item_count))
        print("{}: 关闭了".format(spider.id))