An evaluation of automated methods for cell type annotation in scRNA-seq data

Abstract

Single-cell RNA sequencing (scRNA-seq) is a powerful new method that makes it possible to study gene expression data at the level of individual cells. Cell type annotation, using a reference sets, is a crucial step in this analysis for obtaining insights into tissue and cell composition. However, there is a need to evaluate and objectively know which are the best annotation tools in the immunology field. In this study, we evaluated the performance of four current automatic cell type annotation methods: Support Vector Machine (SVM), SVMrejection, SingleR and scType using three test sets (MCA, PBMCs and JArribas) and two reference sets (ImmGen and Monaco). Overall, the best-performing method was SingleR based on the percentage of correctly classified cells and the weighted-average F1 score. The results also showed that the classification methods were able to correctly predict most of the cells belonging to a cell type, when there was a good representation of this cell type in the test data. Moreover, SVMrejection not only did not improve the results of SVM but it worsened them. Our findings suggest that SingleR is the best annotation tool, especially when it is fitted for each cell using immune data and the reference set is small or the cell types are imbalanced. As SVMrejection did not perform well, other options must be researched in order to annotate when there are no common cell types between test and reference sets.