The castle was founded by the De Lisle family in the 12th century, and then passed to Sir
William de Aldeburgh, following his marriage to Elizabeth de Lisle, heiress of Harewood, who was granted a
licence to crenellate in 1366. When the
second Baron Aldeburgh died in 1391 without issue, the castle transferred to the Ryther and the Redmayne (Redman) families, into which his two daughters had married. In 1574, James Ryther and partner William Plompton bought out the Redman family, although Ryther's financial situation must have worsened because he died in
London's
Fleet Prison in 1595. His son and two daughters sold the castle to Sir William Wentworth of Gawthorpe Hall in 1600 to clear debts; this is probably when Harewood Castle ceased to be a main residence. The castle was last occupied in the 1630s, and in 1656 it was put up for sale as an 'upstanding source of stone and timber'. The Wentworths sold Harewood and Gawthorpe to
Sir John Cutler, 1st Baronet, by which time the castle had probably already been partly dismantled, though Cutler lived there at the end of his life. At Cutler's death in 1693, it passed to his only surviving daughter,
Elizabeth, Countess of Radnor and on her death without heir to Cutler's nephew, unmarried
Edmund Boulter MP of Boston and
Wimpole Hall then to his nephew John Boulter of Gawthorpe and Westminster, who died unmarried in 1738. His executors sold to
Henry Lascelles (1690–1753) whose son,
Edwin Lascelles, 1st Baron Harewood built
Harewood House. Centuries after it had been abandoned, Harewood Castle remained a landmark, the subject of several paintings in the 1797 by
J. M. W. Turner. ==Present==