Search
Abstract
In this work, we improve the generative replay in a continual learning setting to perform well on challenging scenarios. Because of the growing complexity of continual learning tasks, it is becoming more popular, to apply the generative replay technique in the feature space instead of image space. Nevertheless, such an approach does not come without limitations. In particular, we notice the degradation of the continually trained model’s performance could be attributed to the fact that the generated features are far from the original ones when mapped to the latent space. Therefore, we propose three modifications that mitigate these issues. More specifically, we incorporate the distillation in latent space between the current and previous models to reduce feature drift. Additionally, a latent matching for the reconstruction and original data is proposed to improve generated features alignment. Further, based on the observation that the reconstructions are better for preserving knowledge, we add the cycling of generations through the previously trained model to make them closer to the original data. Our method outperforms other generative replay methods in various scenarios. Code available at https://github.com/valeriya-khan/looking-through-the-past.
Citations
-
0
CrossRef
-
0
Web of Science
-
0
Scopus
Authors (5)
-
Valeriya Khan
- Politechnika Warszawska
-
Kamil Deja dr inż.
- Politechnika Warszawska
-
Tomasz Trzciński dr hab. inż.
- IDEAS NCBR
-
Bartłomiej Twardowski dr inż.
- IDEAS NCBR
Cite as
Full text
- Publication version
- Accepted or Published Version
- DOI:
- Digital Object Identifier (open in new tab) 10.1109/ACCESS.2024.3379148
- License
-
open in new tab
Keywords
Details
- Category:
- Articles
- Type:
- artykuły w czasopismach
- Published in:
-
IEEE Access
no. 12,
pages 45309 - 45317,
ISSN: 2169-3536 - Language:
- English
- Publication year:
- 2024
- Bibliographic description:
- Khan V., Cygert S., Deja K., Trzciński T., Twardowski B.: Looking through the past: better knowledge retention for generative replay in continual learning// IEEE Access -Vol. 12, (2024), s.45309-45317
- DOI:
- Digital Object Identifier (open in new tab) 10.1109/access.2024.3379148
- Sources of funding:
-
- Finansowane ze środków IDEAS NCBR
- Verified by:
- Gdańsk University of Technology
seen 48 times
Recommended for you
MagMax: Leveraging Model Merging for Seamless Continual Learning
- D. Marczak,
- B. Twardowski,
- T. Trzciński
- + 1 authors
Revisiting Supervision for Continual Representation Learning
- D. Marczak,
- S. Cygert,
- T. Trzciński
- + 1 authors
Category Adaptation Meets Projected Distillation in Generalized Continual Category Discovery
- G. Rypeść,
- D. Marczak,
- S. Cygert
- + 2 authors
Divide and not forget: Ensemble of selectively trained experts in Continual Learning
- G. Rypeść,
- S. Cygert,
- V. Khan
- + 3 authors