Which tool are most people using incorrectly?

Researchers often misclassify social bots on Twitter

Anyone who tweets a lot is almost a bot

The method has long been controversial within research. First, because classification methods based on machine learning like botometers are only as good as the data with which they are trained. Second, because many researchers seem to like to make things easy for themselves when they use it. Because although the developers themselves warn against using a fixed limit value on their website, for example, most studies do exactly that. For example, it can happen that some authors a result of over 0.43 is sufficient to have an account in their publication as Classify social bot. Other researchers, on the other hand, take 0.76.

So far, this has not done anything to diminish the great media response to relevant studies. Now, however, a current paper clearly shows on the basis of several test series that the general reliability of the instrument simply does not meet social science standards.

To arrive at this result, Adrian Rauchfleisch from the National University of Taiwan and Jonas Kaiser from Harvard University repeatedly fed botometers with five different data sets. Two of the data sets consisted exclusively of verified human Twitter accounts: members of the German Bundestag and those of the US Congress.

Two other lists again only contained bots - but those that for the most part did not disguise their artificiality. These were mostly funny or useful finger exercises by creative programmers, such as those collected on the Botwiki page. As a fifth data set, Rauchfleisch and Kaiser used a mixed list that had been drawn up by the makers of the botometer themselves. The correct diagnosis should have been pretty easy for the program.

The disabled tools of bot research

For three months, the two researchers had their data read out daily by botometer. This showed how imprecise the results delivered were. For an analysis, for example, the authors created a so-called resample of a total of 100,000 accounts from the data of the German parliamentarians and that of the bots, in which they took into account the proportion of bots in the real Twitter community assumed by some researchers - around 15 percent. The new data set contained 85,000 verified human accounts and 15,000 confirmed bots.

In the subsequent calculation, however, 70 percent of the robots found were actually human accounts. At the same time, however, the program missed over 80 percent of the actual bots in the sample - although the authors used the same threshold value for identification as a previously published German bot study. In addition, the results fluctuated depending on the measurement date: on one day individual users were identified as people, on another as bots.