Abergavenny was born at Uckfield House,
Uckfield, Sussex, the second son of
Lord Rupert Nevill (1923–1982) and his wife,
Lady Camilla Anne Evelyn Wallop (1925–2023), daughter of
Gerard Wallop, 9th Earl of Portsmouth. His parents were close friends of the royal family. His grandfather
Guy Larnach-Nevill, 4th Marquess of Abergavenny had died the year before his birth and was succeeded by his uncle
John Nevill, 5th Marquess of Abergavenny (1914–2000). In 1965, his uncle's only son, Henry, Earl of Lewes, died of acute anemia at age 17, making Lord Rupert the heir presumptive to the marquessate. He was educated at
Harrow. ==Later life==