Establishing a framework for the ethical and legal use of web scrapers by cybercrime and cybersecurity researchers: learnings from a systematic review of Australian research.

This article examines the ethical and legal frameworks governing the use of automated data collection technologies (ADCT)—including web crawlers and web scrapers—in Australian cybersecurity research. Through a systematic review of 30 Australian-affiliated studies deploying ADCT across various online domains such as dark web marketplaces, social media, and e-commerce sites, the article identifies key risks including privacy violations, potential breaches of website Terms of Use, copyright infringement, and possession of illegal content. It analyzes these challenges within the context of Australian legislation, notably the Data and Availability and Transparency Act 2022, the Privacy Act 1988, the Copyright Act 1968, and relevant criminal codes, alongside ethical guidelines from the National Statement for Ethical Conduct in Human Research. The article concludes that ADCT use can be both ethical and legal if mitigating measures—such as data de-identification, limiting data collection scope, securing data storage, and obtaining ethical approvals—are implemented, and it offers practical directions to guide researchers navigating this complex and evolving landscape.