This paper shows that there are large differences in cognitive development between children from rich and poor backgrounds at the age of 3, and that this gap widens by the age of 5. Children from poor backgrounds also face much less advantageous "early childhood caring environments" than children from better off families. For example we identify differences in poor children's and their mothers' health and well-being (e.g. birth-weight, breast-feeding, and maternal depression); family interactions (e.g. mother child closeness); the home learning environment (e.g. reading regularly to the child); parenting styles and rules (e.g. regular bed-times and meal-times), and experiences of childcare by ages 3 and 5. Differences in the home learning environment, particularly at the age of 3 have an important role to play in explaining why children from poorer backgrounds have lower test scores than children from better off families. However, a much larger proportion of the gap remains unexplained, or appears directly related to other aspects of family background (such as mothers' age, and family size) that are not mediated through the early childhood caring environment.