He married Lady Elizabeth (or Frances) Carteret, daughter of
John Carteret, 2nd Earl Granville in 1748, and they had four daughters and two sons. The eldest son died in infancy; the younger,
George, became fifth Marquess, and died in 1770 at age 13. The title then devolved on his uncle,
George Hay. Frances was described by Horace Walpole and the Earl of Morton as merry, good company, and able to sing French, Greek, and Scotch songs.
Free Gardener Hay was a
Free Gardener. He was Initiated in the
Dunfermline Lodge of Free Gardeners in 1721 and was immediately made Chancellor (Master) of the Lodge. His interest in the Lodge of Free Gardeners was an abiding one for he was again recorded as being Chancellor of the Lodge in 1742. There is no record of him becoming a Freemason. ==References==