The human long-term memory is astonishingly powerful but fallible at the same time. This makes it very easy to forget information one is sure one actually knows. We propose context-aware augmented memory systems as a solution to this problem. In this paper, we analyse the user interface and interaction design challenges that need to be overcome to build such a system. We hope for fruitful interdisciplinary discussions on how best to address these challenges.