Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies