The castle grew around the Sinweh Tower (c1138-90) as a stronghold of the Holy Roman Empire. Later, it became an imperial residence and Frederick II had a garden made here in the 11th century. He was the last of the Hohenstaufens, a notable family of castle builders.