In the 13th century, the property was known as Aldham and owned by a family of the same name, later passing to the St Clere family. Aldham became Aldham St Cleres and eventually St Cleres. The St Clere family died out in the 1400s and the estate was acquired by Henry Lovell, who built a new house there. In 1625 it passed to Sir John Sedley, 2nd Baronet (of Great Chart), who built the present house. By the death of the 4th baronet in 1702 the house was somewhat neglected and was sold to William Glanville, MP (formerly William Evelyn),
High Sheriff of Kent for 1757. The house passed down in the Evelyn family via
William Evelyn, MP and his son-in-law Alexander, who took the surname Evelyn and was High Sheriff of Kent for 1816. He left it to a cousin Lt-Col William John Evelyn, who rarely used the house and occasionally rented it out and in 1878 it was finally sold to
Sir Mark Wilks Collet, Bt., a London merchant and later a
Governor of the Bank of England. Collet carried out a series of renovations to the buildings and grounds, as did his son and successor Sir Mark Edlman Collet. When the latter moved abroad in 1935 the estate passed to
Montagu Norman, 1st Baron Norman, in whose family it remains. == In popular culture ==