1. Proceedings
  2. I3M
  3. 2019
  4. EMSS
  5. 10.46354/i3m.2019.emss.046

Towards more efficient multiclass AUC computations

  • a Stephan Dreiseitl  
  • a University of Applied Sciences Upper Austria, Campus Hagenberg, Austria
Cite as
Dreiseitl S. (2019). Towards more efficient multiclass AUC computations. Proceedings of the 31st European Modeling & Simulation Symposium (EMSS 2019), pp. 327-331. DOI: https://doi.org/10.46354/i3m.2019.emss.046.

Abstract

The area under the receiver operating characteristics curve (AUC) can be used to assess the discriminatory power of a dichotomous classifier model. Extending this measure to more than two classes is not obvious, and a number of variants have been proposed in the literature. We investigate a heuristic approximation to a method that generalizes the notion of probabilities being correctly ordered, which is equivalent to AUC, to an arbitrary number of classes. While the exact method is computationally complex, we propose a much simpler heuristic that is linear in the number of classes for every combination of data points. Using one artificial and one real-world data set, we demonstrate empirically that this simple heuristic can provide good approximations to the exact method, with Pearson correlation coefficients between 0.85 and 0.998 across all data sets.

References

  1. Bamber D., 1975. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology, 12:387–415.
  2. Bradley A., 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30:1145–1159.
  3. Davis J. and Goadrich M., 2006. The relationship between Precision-Recall and roc curves. In Proceedings of the 23rd International Conference on Machine Learning (ICML-2006), pages 233–240.
  4. Dreiseitl S., Ohno-Machado L., and Binder M., 2000. Comparing three-class diagnostic tests by three-way ROC analysis. Medical Decision Making, 20:323–331.
  5. Edwards D., Metz C., and Kupinski M., 2004. Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Transactions on Medical Imaging, 23:891–895.
  6. Edwards D., Metz C., and Naishikawa R., 2005. The hypervolume under the ROC hypersurface of “nearguessing” and “near-perfect” observers in N-class classification tasks. IEEE Transactions on Medical Imaging, 24:293–299.
  7. Fawcett T., 2006. An introduction to ROC analysis. Pattern Recognition Letters, 27(8):861–874.
  8. Flach P., 2003. The geometry of ROC space: Understanding machine learning metrics through ROC isometrics. In Proceedings of the 20th International Conference on Machine Learning (ICML-2003), pages 226–233.
  9. Fürnkranz J. and Flach P., 2005. ROC ’n’ rule learning—towards a better understanding of covering algorithms. Machine Learning, 58:39–77.
  10. Hand D. and Till R., 2001. A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 45:171–186.
  11. Hanley J. and McNeil B., 1982. The meaning and use of the area under the receiver operating characteristic (ROC) curve. Radiology, 143:29–36.
  12. Hanley J. and McNeil B., 1983. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148:839–843.
  13. He X. and Frey E., 2006. Three-class ROC analysis—the equal error utility assumption and the optimality of three-class roc surface using the ideal observer. IEEE Transactions on Medical Imaging, 25:979–986.
  14. He X. and Frey E., 2008. The meaning and use of the volume under a three-class ROC surface (VUS). IEEE Transactions on Medical Imaging, 28:577–588.
  15. He X., Metz C., Tsui B., Links J., and Frey E., 2006. Three-class ROC analysis—a decision theoretic approach under the ideal observer framework. IEEE Transactions on Medical Imaging, 25:571–581.
  16. Krizhevsky A., 2009. Learning multiple layers of features from tiny images. Technical report, Computer Science Department, University of Toronto.
  17. Landgrebe T. and Duin R., 2007. Approximating the multiclass roc by pairwise analysis. Pattern Recognition Letters, 28:1747–1758.
  18. Lasko T., Bhagwat J., Zhou K., and Ohno-Machado L., 2005. The use of receiver operating characteristic curves in biomedical informatics. Journal of Biomedical Informatics, 38(5):404–415.
  19. Lusted L., 1978. General problems in medical decision making with comments on ROC analysis. Seminars in Nuclear Medicine, 8:299–306.
  20. Metz C., 1978. Basic principles of ROC analysis. Seminars in Nuclear Medicine, 8:283–298.
  21. Mossman D., 1999. Three-way ROCs. Medical Decision Making, 19:78–89.