Structural origins of catastrophic forgetting in self-supervised continual learning: A directional and curvature-based analysis of the learning signal

Abstract

Catastrophic forgetting remains a fundamental challenge in continual learning, where acquiring new knowledge systematically degrades previously learned representations. While existing approaches primarily mitigate this by imposing architectural constraints or using data replay strategies, they offer limited theoretical insight into why and how parameter updates interfere with consolidated knowledge structures.

This thesis proposes a structured analytical lens to examine the collision between new and old knowledge at the level of parameter updates, feature representations, and low-dimensional learning signals. Rather than introducing a new method, we seek to characterize the geometric conditions under which gradient updates pose the greatest risk to previously learned structure. Specifically, we hypothesize that forgetting is governed by the degree to which parameter updates project onto high-curvature regions of the old task's loss landscape, namely those directions along which the old loss function is most sensitive to perturbation. We derive a theoretical bound that isolates this curvature-projection term as the dominant factor driving representational forgetting, and empirically verify both the structural conditions under which this bound holds and its statistical relationship with observed forgetting.

To ground this analysis in a concrete and mechanistically interpretable setting, we adopt SwAV, a representative self-supervised contrastive learning framework, as our experimental substrate. We leverage SwAV's internal prototype assignment process as a low-dimensional learning signal that faithfully reflects the underlying representational dynamics, allowing the theoretical bound to be expressed and studied in a tractable, interpretable form.

Building on this, we further consider whether the second-order sensitivity structure of old knowledge, when projected into a lower-dimensional subspace, retains meaningful geometric differentiation between sensitive and insensitive directions. Our experiments confirm that such a structure persists at low dimensionality and that SwAV's learning signal selectively engages it. We observe that interference with prior knowledge is measurably reduced when the energy of the new task's low-dimensional signal concentrates along axes that carry less of the old task's curvature structure, rather than along regions of high coupling.