Milton married, on 19 April 1933, Olive Dorothea "Obby" Plunket (died 1975), a daughter of
Benjamin Plunket,
Bishop of Tuam, Killala and Achonry, and a granddaughter of
William Plunket, 4th Baron Plunket, who was
Archbishop of Dublin. They had one daughter,
Lady Anne Juliet Dorothea Maud Wentworth-Fitzwilliam, born on 24 January 1935. In 1943, he inherited the Earldom and a great fortune in land, houses, and art, from his father. In Lord Fitzwilliam's later years, his marriage was in disarray, partly due to Olive's
alcoholism, and at the time of his death he was seeking a divorce in order to marry someone else. From 1946, he had been romantically linked with the widowed
Kathleen Cavendish, Marchioness of Hartington, sister of the future U.S. President
John F. Kennedy. She was killed with him in an air crash on 13 May 1948, although the nature of their relationship was not made clear in the newspaper accounts at the time. On his death, leaving no son, Fitzwilliam's peerages passed to his second cousin once removed,
Eric Spencer Wentworth-Fitzwilliam, but his fortune, then estimated at £45 million, including half of the Wentworth Woodhouse estate, the
Coollattin estate in
County Wicklow, Ireland, and a large part of the Fitzwilliam art collection, were inherited by his thirteen-year-old daughter, the present
Lady Juliet Tadgell. ==In popular culture==