Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors

We propose a robustness testing approach for software systems that process large amounts of data. Our method usesmetamorphic relations to check software output for erroneous input in the absence of a tangible test oracle. We use this technique to testtwo major citation database systems: Scopus and the Web of Science. We report a surprising finding that the inclusion of hyphensin paper titles impedes citation counts, and that this is a result of the lack of robustness of the citation database systems in handlinghyphenated paper titles. Our results are valid for the entire literature as well as for individual fields such as chemistry. We further finda strong and significant negative correlation between the journal impact factor (JIF) of IEEE Transactions on Software Engineering(TSE) and the percentage of hyphenated paper titles published in TSE. Similar results are found for ACM Transactions on SoftwareEngineering and Methodology. A software engineering field-wide study reveals that the higher JIF-ranked journals are publishing a lowerpercentage of papers with hyphenated titles. Our results challenge the common belief that citation counts and JIFs are reliable measuresof the impact of papers and journals, as they can be distorted simply by the presence of hyphens in paper titles.