Technical report TR06-25. In this paper we present a mathematical foundation for Incremental Least-Squares Temporal Difference Learning (iLSTD) for policy evaluation in reinforcement learning with linear function approximation. iLSTD is an incremental method for achieving results similar to...