Charles O Dexter established the garden. He was a famous rhododendron collector and hybridizer and worked with Charles Sprague Sargent on the Arnold Arboretum collection. The estate was then bought by Josiah Lilly and used to established a collection of Americana - cultural artefacts. Lilly also restored and extended the garden, creating a collection of day lilies. In Autumn 2006, fifty thousand daffodils and tulips were planted.