Lifelong Augmentation of Multi-Modal Streaming Autobiographical Memories

Tobias Fischer*, Maxime Petit*, and Yiannis Demiris
*: equal contributions
Imperial College London
Personal Robotics Lab

Framework overview

Framework overview


Many robotics algorithms can benefit from storing and recalling large amounts of accumulated sensorimotor and interaction data. We provide a principled framework for the cumulative organisation of streaming autobiographical data so that data can be continuously processed and augmented as the processing and reasoning abilities of the agent develops and further interactions with humans take place. As an example, we show how a kinematic structure learning algorithm reasons a-posteriori about the skeleton of a human hand. A partner can be asked to provide feedback about the augmented memories, which can in turn be supplied to the reasoning processes in order to adapt their parameters. We employ active, multi-modal remembering, so the robot as well as humans can gain insights of both the original and augmented memories. Our framework is capable of storing discrete and continuous data in real-time, and thus creates a full memory. The data can cover multiple modalities and several layers of abstraction (e.g. from raw sound signals over sentences to extracted meanings). We show a typical interaction with a human partner using an iCub humanoid robot. The framework is implemented in a platform-independent manner. In particular, we validate multi platform capabilities using the iCub, Baxter and NAO robots. We also provide an interface to cloud based services, which allow automatic annotation of episodes. Our framework is geared towards the developmental robotics community, as it 1) provides a variety of interfaces for other modules, 2) unifies previous works on autobiographical memory, and 3) is licensed as open source software.


Open Access Download
 Link to IEEE Xplore



 Source Code Download

If you want to use the source code, you need to clone the WYSIWYD repository. Examples for the usage can be found here (ABMAugmentionExample) and here (ROStoYarpBridge).


 BibTeX Download


This research was funded in part by the EU projects WYSIWYD (grant FP7-ICT-612139) and PAL (grant H2020-PHC-643783). The authors thank the members of the Personal Robotics Lab for their continued support, particularly Hyung Jin Chang (providing the kinematic structure learning module) and Martina Zambelli for helping with the human-robot interaction.