On 1 September 1767 at St George's in
Hanover Square, London, he married Frances Lambertine Christiana Charlotte Harriet Theresa de Kolbel, the daughter of a German
nobleman, without issue. He owned Carlisle's Estate in St George,
Antigua, with over 300 slaves but he died almost destitute and the estate was bought by Reid and Dixon. Payne was involved in no claim for compensation at the time of abolition of slavery. His wife was well connected and mixed in high social circles, including with
Queen Charlotte. Payne therefore made good contacts for his political career through his marriage; however, his time with Frances was not a particularly happy one. He owned Carlisle's Estate in St George,
Antigua, with over 300 slaves but he died almost destitute and the estate was bought by Reid and Dixon. Payne was involved in no claim for compensation at the time of abolition of slavery. He died at Government House, Leeward Islands, on 3 August 1807. He was buried the next day at his plantation, Carlisles, and a marble monument to him was erected in the parish church of St John's in Antigua. He died without children and in a dire economic situation, leaving his wife, Lady Lavington, to sustain herself on an annual income of just £300; an extremely low amount for the widow of a peer and relatively influential politician, although it would have allowed her to keep a servant. Lady Lavington died at
Hampton Court Palace in London, UK on 2 May 1830. According to biographer W.P. Courtney, Lord Lavington's "career mirrored the meteoric rise and downfall of absentee sugar planters in Britain." ==Styles and honours==