The Hermitage was originally built in 1830 by the Reverend George Sheed. The property is about two miles west of Ancaster, in the Dundas Valley.

Otto Ives (1804–1835) was the third landowner. He was an English officer who had fought in the Greek War of Independence against the Ottoman Empire.

He met Magdalene Diamanti, a daughter of the Governor of an Aegean island. They married in Corfu in 1824 and had arrived in Ancaster by 1833.
They brought her sister or niece with them to act as a companion for Mrs. Ives. It was here that Otto Ives purchased the Hermitage from the heirs of the Rev. George Sheed. Ives had hired a coachman by the name of William Black.

Although it is said that Black was also a tutor in the English language, evidence exists that this post was filled by Mary Rosebeury
(later Mrs. Peter Filman of Hamilton) and it is said that Black fell in love with the sister or niece.

Black went to Otto Ives and asked for his niece's hand in marriage. Ives was very upset by the thought and rejected the proposal.

The next morning, Ives and his wife were to go out for the day, but the coachman was not at the front door with the carriage as planned. Ives went out to the barn to see why the coachman had not appeared and discovered Black's body dangling from the rafters in the barn near the first Hermitage.