The island rule has been widely applied to a range of taxonomic groups, with some studies reporting supporting evidence but others questioning this hypothesis. To bring more clarity to this debate, we conducted a comparative analysis of the available literature, focussing on potential biases. Worldwide. We performed a systematic review to identify studies testing the island rule and translated these studies’ outcomes, so that they follow a consistent approach. The studies were assessed for differences in their analysis of the island rule. We created an authorship network showing who published studies with whom on the topic and weighted the data based on co-authorship and number of publications. We identified 143 relevant studies, finding a significantly lower frequency of supporting studies according to our consistent approach (50%) than the authors’ own statements (59%). Two core-author groups could be identified with a strong publication record on the island rule. The first group has predominately published studies supporting the rule, whereas the other group has mainly published studies questioning it. According to a subsequent analysis excluding studies with a high risk of HARKing (hypothesizing after the results are known), the frequency of studies supporting the rule further dropped to 42%. Empirical support for the island rule is low, especially for non-mammalian taxa and when using a consistent evaluation approach. Differences among studies in supporting versus questioning this hypothesis seem to be partly due to author-related biases. Methods to address potential biases in studying ecological hypotheses are urgently needed. We offer such a method here.