Duff lived at
Delgatie Castle near Turriff, Aberdeenshire. He had married Anne Stein, daughter of James Stein of Kilbagie and Kennetpans House and sister to
John Stein, MP for
Bletchingley. by whom he had two sons and two daughters, including: •
James Duff, 5th Earl Fife (1814–1879), who married
Lady Agnes Georgiana Elizabeth Hay, daughter of
William Hay, 18th Earl of Erroll and
Lady Elizabeth FitzClarence (an illegitimate daughter of
William IV). •
George Skene Duff (1816–1889), an MP for
Elgin Burghs. • Catherine Duff (–1869), who married
John Lewis Ricardo, MP for
Stoke-upon-Trent, and nephew of economist
David Ricardo, in 1841. He died at Percy Cross, Walham Green, Fulham, Middlesex in March 1851.
Freemasonry He was Initiated into Scottish Freemasonry in Lodge Holyrood House (St Luke's), No.44, on 30 November 1812. ==References==