|
| | | | |
| | | |
| Stat |
Members: 3645
Articles: 2'506'133
Articles rated: 2609
26 April 2024 |
|
| | | | |
|
Article overview
| |
|
|
A robust estimator of mutual information for deep learning interpretability | | Davide Piras
; Hiranya V. Peiris
; Andrew Pontzen
; Luisa Lucie-Smith
; Ningyuan Guo
; Brian Nord
; | | Date: |
31 Oct 2022 | | Abstract: | We develop the use of mutual information (MI), a well-established metric in
information theory, to interpret the inner workings of deep learning models. To
accurately estimate MI from a finite number of samples, we present GMM-MI
(pronounced $’’$Jimmie$"$), an algorithm based on Gaussian mixture models that
can be applied to both discrete and continuous settings. GMM-MI is
computationally efficient, robust to the choice of hyperparameters and provides
the uncertainty on the MI estimate due to the finite sample size. We
extensively validate GMM-MI on toy data for which the ground truth MI is known,
comparing its performance against established mutual information estimators. We
then demonstrate the use of our MI estimator in the context of representation
learning, working with synthetic data and physical datasets describing highly
non-linear processes. We train deep learning models to encode high-dimensional
data within a meaningful compressed (latent) representation, and use GMM-MI to
quantify both the level of disentanglement between the latent variables, and
their association with relevant physical quantities, thus unlocking the
interpretability of the latent representation. We make GMM-MI publicly
available. | | Source: | arXiv, 2211.00024 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
|
| |
|
|
|
|
News, job offers and information for researchers and scientists:
| |
REGISTER