His sister was Rose Whitworth-Aylmer who travelled with their aunt, Anne Barbara, and her husband
Sir Henry Russell, 1st Baronet to India in 1798. She fell ill with cholera and died in 1800. She has been immortalized by his friend and the famous British poet
Walter Savage Landor in his Rose Aylmer eulogy. General the Right Honourable Matthew Whitworth, 5th Lord Aylmer, G.C.B., married Louisa Anne Call, daughter of Sir John Call, Bart. on 4 August 1801. When her husband was appointed in 1830 to administer the government of Canada, as Governor-General, from February 1831 to August 1835, the couple entertained at the Castle of St. Louis, Quebec. During the cholera epidemic of 1831–32, she was interested in the relief of the sufferers. Since she was interested in education, she regularly visited and bestowed prizes in the schools. She served as the patroness of the Societe d'Education sous la direction des dames de Quebec. She died on 13 August 1862. ==References==