Potter married Elizabeth Palmer, daughter of John Westorby Palmer, of York in January 1808. Elizabeth had two children, Elizabeth and Anne, before she died in February 1810. After her death, John married Esther Bayley of Booth Hall, Blackley in Manchester in September 1812 in the
Collegiate Church. They had four children; Esther (born July 1813 and died February 1814),
John, Mary (born October 1816 and died March 1817) and
Thomas. Esther senior died in June 1852 at Buile Hill in Salford and was buried in Ardwick Cemetery in June 1852. In 1818 she founded Lady Potter's Schools at
Irlams o' th' Height enabling 80 girls to receive an education. In 1825 Thomas Potter commissioned the architect, Sir Charles Barry, to design him a house at Buile Hill in Salford. This is the only house where Barry used Greek revival architecture. It was completed in 1827. After the death of Sir Thomas's first son, Sir John Potter, Buile Hill was inherited by Sir Thomas's second son, Thomas Bayley Potter, whose youngest son, Richard Ellis Potter, was born there on 3 October 1855. Thomas Bayley Potter sold the property to John Bennett in 1877. Salford Corporation purchased the estate in 1902 and in 1906 the house opened as Salford Natural History Museum. The 80 acres of
parkland were opened to the public. Buile Hill House is a grade II listed building and is one of the case studies of the Georgian Group who advocate that the principal reception rooms, staircase and hall should remain as they are (with restoration). There has been some talk of Salford City Council selling it off for hotel use but there is also a movement for its retention pending a further decision. ==Death==