In 1523 the castle was already described as not useful for defence purposes but with thin walls. The repair by the Italian architect Licio de Spari 1538 - 41 for the building owner, the Augsburg patrician Paumgarten, was nearly a rebuilding.
After repeated change of ownership, the Crown Prince Maximilian of Bavaria was addicted to the charm of the place and bought the meanwhile to ruin decayed castle. The restoration started in 1833.
King Maximilian II has used the castle as summer residence for his family with his son the later King Ludwig II.
