A Novel Cooperation and Competition Strategy Among Multi-Agent Crawlers

Year, pages: 2016, 1050 - 1078

Multi-Agent theory which is used for communication and collaboration among focused crawlers has been proved that it can improve the precision of returned result significantly. In this paper, we proposed a new organizational structure of multi-agent for focused crawlers, in which the agents were divided into three categories, namely F-Agent (Facilitator-Agent), As-Agent (Assistance-Agent) and C-Agent (Crawler-Agent). They worked on their own responsibilities and cooperated mutually to complete a common task of web crawling. In our proposed architecture of focused crawlers based on multi-agent system, we emphasized discussing the collaborative process among multiple agents. To control the cooperation among agents, we proposed a negotiation protocol based on the contract net protocol and achieved the collaboration model of focused crawlers based on multi-agent by JADE. At last, the comparative experiment results showed that our focused crawlers had higher precision and efficiency than other crawlers using the algorithms with breadth-first, best-first, etc.

Du, Y., Xu, Y., Wang, M. 2016. A Novel Cooperation and Competition Strategy Among Multi-Agent Crawlers. In Computing and Informatics, vol. 35, no.5, pp. 1050-1078. 1335-9150.

Du, Y., Xu, Y., Wang, M. (2016). A Novel Cooperation and Competition Strategy Among Multi-Agent Crawlers. Computing and Informatics, 35(5), 1050-1078. 1335-9150.