Lord Gainsborough married Lady Ida Harriet Augusta, daughter of
William Hay, 18th Earl of Erroll and
Elizabeth FitzClarence, illegitimate daughter of
King William IV, in 1841. The family home was at
Flitteriss Park. They had five children: • Lady Blanche (25 March 1845 – 21 March 1881), who married Thomas Murphy • The "Lady Blanche House" is commemorated by
New Hampshire Historical Marker No. 109 in
Bartlett, New Hampshire • Lady Constance (19 October 1847 – 8 April 1891), who married
Sir Henry Bellingham, 4th Baronet •
Charles, 3rd Earl of Gainsborough (20 October 1850 – 17 April 1926), had issue • Edward (28 April 1852 – 9 November 1917), who married Ruth Lucas • Lady Edith (1849 - 22 August 1890), who became a nun Lady Gainsborough died in October 1867. Lord Gainsborough remained a widower until his death in August 1881, aged 62. He was succeeded in the earldom by his eldest son,
Charles. His grandsons included
Edward Noel and
John Baptist Lucius Noel. ==References==