Robust Evaluation of Neural Networks for Contrail Detection

Accurate detection of aircraft contrails from satellite imagery is a valuable capability for studying one of the most impactful yet uncertain climate effects of aviation. In a recent study published in IEEE Transactions on Geoscience and Remote Sensing, Irene Ortiz Abuja and co-authors critically assess the performance of state-of-the-art deep learning models trained on the OpenContrails dataset, with the aim of explaining the segmentation performance plateau widely reported in the literature.

The study evaluates an ensemble of six neural network architectures and reports several key findings. For segmentation, performance exhibits an upper bound of 88% Global Dice Score (GDS), with the proposed ensemble already operating near this theoretical limit. The analysis further shows that reported performance metrics are strongly influenced by annotation uncertainty at contrail boundaries, and that evaluations based on rigid ground truths—without tolerance for label ambiguity—tend to obscure true model segmentation capability.

In terms of detection, the ensemble identifies 93% of target contrail features while maintaining a false positive rate below 3%. The most challenging scenarios arise in scenes containing thick ice clouds, small or isolated contrail segments, and aged, diffused contrails.

Main conclusions

(a) Imperfect and inconsistent annotations bias evaluation metrics and mask the true segmentation performance of current models.

(b) Limited training data for aged and diffuse contrails restricts generalization beyond young, linear contrail cases.

(c) Future progress is more likely to come from improvements in datasets, annotation consistency, and evaluation methodologies than from further neural network architectural refinements.

Overall, this work reinforces a central conclusion of the E-CONTRAIL initiative: meaningful advances in contrail detection and segmentation will require prioritizing data quality and robust evaluation strategies over continued increases in model complexity.