𝕏-scraper

Logo

(Version française)

An extension for extracting and downloading tweets for text mining.

Cite this program

If you use this extension for your research, please reference it as follows:

Moncomble, F. (2024). 𝕏-Scraper (Version 0.4) [JavaScript]. Arras, France: Université d’Artois. Available at: https://fmoncomble.github.io/X-scraper/

Installation

Firefox

Firefox add-on

Chrome/Edge

available-chrome-web-store4321

Remember to pin the add-on to the toolbar.

Instructions for use

Known issues and limitations

Too many requests

The add-on collects tweets by automatically scrolling the search results page. This makes repeated calls to the 𝕏/Twitter server, which eventually times out with a 429 response (Too many requests). When that happens (generally after scraping ~900 tweets), download the file, reset, allow a few minutes for the server to ‘cool down’, then adjust your search parameters to avoid collecting duplicates and resume scraping.

Interface redesign

⚠️ Important! In v0.2, the add-on’s popup window needs to remain open for the extension to behave properly. Clicking outside it, switching to another tab/window, or switching to a different app will cause it to close, effectively preventing the user from interacting with the extension during or after the scraping process.
This is addressed in v0.3 through a redesigned interface: make sure to download the newest version.

Create an ad-hoc account

Although Elon Musk has repeatedly expressed his opposition to scraping 𝕏/Twitter data, collecting publicly available data for research purposes is legal in most countries. However, as a precaution, it is advisable to create an ad-hoc account for this specific purpose.