Methods for Image Data Analysis
The MIDA group focuses on development of general methods for image data analysis. Our aim is to devise generally applicable methods, which work well independent of the particular application and types of images used. We therefore strive for robust methods which are performing well under varying conditions. Also aiming for practically useful methods, we essentially always collaborate with other groups, including Social Robotics Lab, Quantitative Microscopy and MedIP - Medical Image Processing.
- Explainable Artificial Intelligence
- Image Registration
- Representation Learning and Image Translation
- Image Similarity and Distance Measures
Explainable Artificial Intelligence
Interpretation of classification behaviour of deep neural network models
Deep convolutional neural networks demonstrate state-of-the-art performance in many image analysis tasks, however, their opacity does not allow to infer how they arrive at a decision. We are aiming at detection of oral cancer at an early stage, and it is particularly important to develop a reliable algorithm. In our workflow, trained deep convolutional neural networks are used to differentiate cytological images into normal and abnormal classes. We examine methods that could elevate understanding of the deep learning classification properties and enable interpretation of data classification. Furthermore, we would like to increase understanding of the premalignant state by exploring and visualizing what parts of cell images are considered as most important for the task.
- M. Fraile, J. Lindblad, C. Fawcett, N. Sladoje, and G. Castellano. Automatic analysis of infant engagement during play: An end-to-end learning and Explainable AI pilot experiment. In Proceedings of ICMI 2021 Workshop on Bridging Social Sciences and AI for Understanding Child Behavior (WoCBU)
- N. Koriakina, N. Sladoje, E. Wetzer, J. Lindblad. Uncovering hidden reasoning of convolutional neural networks in biomedical image classification by using attribution methods. 4th NEUBIAS Conference, Bordeaux, France, March 2020.
- N. Koriakina, N. Sladoje, E. Bengtsson, E. Darai Ramqvist, J-M. Hirsch, C. Runow Stark, J. Lindblad. Visualization of convolutional neural network class activations in automated oral cancer detection for interpretation of malignancy associated changes. 3rd NEUBIAS Conference, Luxembourg, Feb. 2019.Poster
Trustworthy AI-based decision support in cancer diagnostics
To reach successful implementation of AI-based decision support in healthcare it is important to have trust in the system outputs. One reason for lack of trust is the lack of interpretability of the complex non-linear decision making process. A way to build trust is thus to improve humans’ understanding of the process, which drives research within the field of Explainable AI. For a successful implementation of AI in healthcare and life sciences, it is imperative to acknowledge the need for cooperation of human experts and AI-based decision making systems: Deep learning methods, and AI systems, should not replace, but rather augment clinicians and researchers. This project aims to facilitate understandable, reliable and trustworthy utilization of AI in healthcare, empowering the human medical professionals to interpret and interact with the AI-based decision support system. Watch here an overview of how deep learning can aid screening programs for early oral cancer detection, presented at IT20, the 20th anniversary celebrations of Uppsala University's IT Department [Video].
- N. Koriakina, J. Lindblad, and N. Sladoje. The Effect of Within-Bag Sampling on End-to-End Multiple Instance Learning. In Proceedings of the 12th IEEE International Symposium on Image and Signal Processing and Analysis (ISPA)
- J. Lu, N. Sladoje, C. Runow Stark, E. Darai Ramqvist, J-M. Hirsch, J. Lindblad. A Deep Learning based Pipeline for Efficient Oral Cancer Screening on Whole Slide Images. In Proc. of the Intern. Conf. of Image Analysis and Recognition, ICIAR 2020, Lecture Notes in Computer Science - LNCS 12132, pp 249-261, Springer 2020. [Online]
In this project we study general-purpose registration methods based on distances incorporating both intensity and spatial information. Thus far, we have developed methods that increase convergence regions when using gradient-based optimization, while improving the accuracy, as well as having similar or shorter run-time than comparable intensity-based methods. Through our work on the CoMIR representations, we have shown that these methods can be extended from mono-modal to multi-modal scenarios. Our current research in this project aims to find paths towards greater generality in applicability, increased computational efficiency, and methods to overcome the idiosyncratic challenges of various applications (such as local discontinuities, missing structures, noise and other corruption).
- J. Öfverstedt, J. Lindblad, and N. Sladoje. Fast computation of mutual information in the frequency domain with applications to global multimodal image alignment. Pattern Recognition Letters, Vol.159, pp. 196 - 203, 2022.[Online][Preprint]
- J. Öfverstedt, J. Lindblad, and N. Sladoje. Cross-Sim-NGF: FFT-Based Global Rigid Multimodal Alignment of Image Volumes using Normalized Gradient Fields. 10th Workshop on Biomedical Image Registration, Munich, Germany, 2022.
- J. Öfverstedt, J. Lindblad, and N. Sladoje. Fast Computation of Mutual Information with Application to Global Multimodal Image Alignment of Micrographs. COMULIS Conference 2021 Poster
- J. Öfverstedt, J. Lindblad, and N. Sladoje. INSPIRE: Intensity and Spatial Information-Based Deformable Image Registration. [Preprint]
- J. Öfverstedt, J. Lindblad, N. Sladoje. Fast and Robust Symmetric Image Registration Based on Distances Combining Intensity and Spatial Information. IEEE Trans. on Image Processing, Vol.28(7), pp. 3584 - 3597, 2019. [Online]
Representation Learning and Image Translation
Robust learning of geometric equivariances
The project builds on, and extends recent works on geometric deep learning and aims at combining it with Manifold learning, to produce truly learned equivariances without the need for engineered solutions and maximize benefits of shared weights (parameters to learn). A decrease of the numbers of parameters to learn leads to increased performance, generalizability and reliability (robustness) of the network. An additional gain is in reducing a risk that the augmented data incorporates artefacts not present in the original data. A typical example is textured data, where interpolation performed in augmentation by rotation and scaling unavoidably affects the original texture and may lead to non-reliable results. Reliable texture-based classification is, on the other hand, in many cases of high importance in biomedical applications.
- K. Bengtsson Bernander, J. Lindblad, R. Strand, I. Nyström. Replacing data augmentation with rotation-equivariant CNNs in image-based classification of oral cancer. In Proceedings of the 25th Iberoamerican Congress on Pattern Recognition (CIARP), 2021.
Multimodal Representation Learning
Multimodal images refer to images produced by multiple imaging techniques, such as different sensors. Combining the information of different imaging modalities offers complimentary information about the properties of the imaged specimen. Often these modalities need to be captured in different machines, such that the resulting images need to be registered in order to map the corresponding signals to each other. This can be a very challenging task due to the varying appearance of the specimen in different sensors.
We use contrastive learning to find representations of both modalities which are abstract and very similar with respect to a similarity measure, such that the images of different modalities are mapped into the same representational space. There are requirements that these representations need to fulfil for the downstream task, such as rotational equivariance or intensity similarity, which can be enforced through modifications of the contrastive loss. Finally, in this abstract space, common methods for monomodal registration (intensity-based as well as feature-based) can be used to align the corresponding images. The transformation found this way can be applied to the original images, which overcomes the problem of multimodal registration.
- E. Breznik, E. Wetzer, J. Lindblad, N. Sladoje. Cross-Modality Sub-Image Retrieval using Contrastive Multimodal Image Representations. [Preprint]
- E. Wetzer, N. Pielawski, E. Breznik, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, N. Sladoje. Contrastive Learning for Equivariant Multimodal Image Representations. "The Power of Women in Deep Learning" Workshop at the "Mathematics of Deep Learning" Programme at the Isaac Newton Institute 2021 Poster
- E. Wetzer, N. Pielawski, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, N. Sladoje. Registration of Multimodal Microscopy Images using CoMIR – learned structural image representations. COMULIS Conference 2021 Poster
- N. Pielawski, E. Wetzer, J. Öfverstedt, J. Lu, C. Wählby, J. Lindblad, N. Sladoje. CoMIR: Contrastive Multimodal Image Representation for Registration. In Proc. of NeurIPS 2020 [Online]Poster [Video]
- E. Wetzer, N. Pielawski, J. Öfverstedt, J. Lindblad, I. Floroiu, A. Dumitru, M. Costache, R. Hristu, S. G. Stanciu, N. Sladoje. Cross-modal Representation Learning for Efficient Registration of Multiphoton and Brightfield Microscopy Images of Skin Tissue. 4th NEUBIAS Conference, Bordeaux, March 2020
Generative Adversarial Networks (GANs) can be used to create mappings from image domain A to B, such that the distribution of the mapped, -or translated- images of domain A are indistinguishable from the distribution of B. This approach can be used for style transfer and in consequence to translate from one imaging modality to another; or for data augmentation, in particular in fields in which labelled data is expensive, such as biomedical applications.
- J. Lu, J. Öfverstedt, J. Lindblad, and N. Sladoje. Image-to-Image Translation in Multimodal Image Registration : How Well Does It Work? COMULIS Conference 2021. Poster
- T. Majtner, B. Bajic, J. Lindblad, N. Sladoje, V. Blanes-Vidal, E. S. Nadimi. On the Effectiveness of Generative Adversarial Networks as HEp-2 Image Augmentation Tool. In Proceedings of the Scandinavian Conference on Image Analysis, SCIA2019, Norrköping, Sweden, Lecture Notes in Computer Science, LNCS-11482, pp. 439-451, Springer 2019. [Online]
Multi-layer object representations for texture analysis
Texture features such as local binary patterns have shown to provide complementary information to that of plain intensity images in learning algorithms. We investigate methods on the fusion of texture and intensity sources, as well as the problems connected to the fact that many texture descriptors are unordered sets and require suitable (dis-)similarity measures in order to be processes by for example convolutional neural networks. We develop strategies to integrate more complex texture features into learning methods and evaluate their performance on various biomedical images. Such hybrid object representations show promising results in, e.g., detection and segmentation of high resolution transmission electron microscope (TEM) images, taking it one step closer to automation of pathological diagnostics.
- E. Wetzer, J. Gay, H. Harlin, J. Lindblad, and N. Sladoje. When texture matters: Texture-focused CNNs outperform general data augmentation and pretraining in Oral Cancer Detection. In Proceedings of the 17th IEEE International Symposium on Biomedical Imaging (ISBI), IEEE, pp. 517-521, Iowa City, USA, April 2020. [Online]
- E. Wetzer, J. Gay, H. Harlin, J. Lindblad, and N. Sladoje. Texture-Based Oral Cancer Detection: A Performance Analysis of Deep Learning Approaches. 3rd NEUBIAS Conference, Luxembourg, Feb. 2019.Poster
- B. Bajic, T. Majtner, J. Lindblad, N. Sladoje. Generalized Deep Learning Framework for HEp-2 Cell Recognition Using Local Binary Pattern Maps. IET Image Processing, Vol. 14, No. 6, pp. 1201-1208, 2020. [Online]
- E. Wetzer, J. Lindblad, I.-M. Sintorn, K. Hultenby, N. Sladoje. Towards automated multiscale imaging and analysis in TEM: Glomerulus detection by fusion of CNN and LBP maps. In Proc. of the ECCV 2018, Workshop on BioImage Computing, Lecture Notes in Computer Science, LNCS-11134, pp. 465-475, Munich, Germany, Sept. 2018. [Online] Poster
Image Similarity and Distance Measures
Stochastic Distance Transform
Distance transforms (DTs) are based on minimal distances and are thus noise sensitive; a single noisy point can change the distance substantially. The Stochastic Distance Transform (SDT) is a distance transform based on stochastic modelling of the binary images using the theory of discrete random sets. In this project, we explore theoretical properties of the SDT, efficient algorithms that enable the computation of these distance transforms, and how the accuracy of the resulting distances are substantially improved by adopting the SDT in favor of classical DTs.
- J. Öfverstedt, J. Lindblad, and N. Sladoje. Stochastic Distance Transform: Theory, Algorithms and Applications. Journal of Mathematical Imaging and Vision 62(5), 751-769, 2020. [Online]
- J. Öfverstedt, J. Lindblad, N. Sladoje. Stochastic Distance Transform. In Proceedings of the 21th International Conference on Discrete Geometry for Computer Imagery, DGCI2019, Paris, France, Lecture Notes in Computer Science, LNCS-11414, pp. 75-86, Springer 2019. [Online]
Combining Shape and Intensity Information
Similarity (or distance) measures between images are fundamental components in image analysis, and are used in many tasks such as
- template matching,
- image registration,
- objective functions for training various types of Neural Networks.
We study measures which combine image intensity and spatial information efficiently and aim to demonstrate that they lead to practical, robust, high performance methods for these and other common tasks.
- J. Öfverstedt, N. Sladoje, and J. Lindblad. Distance Between Vector-valued Fuzzy Sets based on Intersection Decomposition with Applications in Object Detection. In Proc. of the 13th International Symposium on Mathematical Morphology, ISMM2017, Fontainebleau, France, Lecture Notes in Computer Science, LNCS-10225, pp. 395-407, Springer 2017. [Online]
- N. Sladoje and J. Lindblad. Distance Between Vector-valued Representations of Objects in Images with Application in Object Detection and Classification. In Proc. of the 18th International Workshop on Combinatorial Image Analysis, IWCIA2017, Plovdiv, Bulgaria, Lecture Notes in Computer Science, LNCS-10256, pp. 243-255, Springer 2017. [Online]
Karl Bengtsson Bernander
Marc Fraile Fabrega
Guest Doctoral Student