A corpus-based approach to identifying stopwords in multiple languages

Authors

  • Bronwyn Bowles-King
  • Tanja Gaustad
  • Muzi Matfunjwa
  • Nomsa Skosana
  • Benito Trollip
  • Michelle White
  • Menno Van Zaanen

DOI:

https://doi.org/10.55492/dhasa.v7i02.7691

Keywords:

Stopwords, South African languages, statistical features

Abstract

Stopwords are generally considered common, high-frequency words that carry little semantic information, and they are often filtered out during preprocessing, for example, to save computational power, improve model performance, or support focused analysis. However, not all languages have a stopword list readily available. This study identifies statistical word features in general-purpose corpora to evaluate existing Afrikaans, English, isiZulu, and Sesotho stopword lists. We consider whether stopword features differ for these languages compared to features typically used to identify English stopwords, such as word length, word frequency, document frequency, and combinations of these features. Although stopwords are ubiquitous in the corpora, they are a tiny minority group in terms of the number of unique words. This skewed distribution raises challenges in statistical modelling as it biases the classifier towards the larger group of content words. We therefore applied elastic net logistic regression with feature ablation for 25 features. The most effective class differentiation was achieved using a reduced trivariate model including word length, log term frequency, and log document frequency, with AUCs between 98.20 and 99.99 for the four languages. Out-of-fold evaluations for this core model demonstrated low precision (0.87--6.14) and F1-scores (1.73--11.53), but high recall (91.36--100.00). Our findings suggest that these three statistical measures for stopword identification behave consistently across the target languages, regardless of orthography or corpus size, and they display the potential to filter candidate stopwords for subsequent evaluation and curation.

Downloads

Published

2026-11-20

How to Cite

A corpus-based approach to identifying stopwords in multiple languages. (2026). Journal of the Digital Humanities Association of Southern Africa (DHASA), 7(2), 56-66. https://doi.org/10.55492/dhasa.v7i02.7691