Chandler's first wife, Mary, died in 1832. On July 3, 1833, he married Anna Maria Holton Jones, a
Catholic originally from
Baltimore. Raised as a
Baptist, Chandler converted to the Catholic Church after the 1852 election. His conversion is cited by
Saint Joseph's University professor of history Frank Gerrity as the reason for his failed reelection in 1854. Prior to joining the Catholic church, Chandler was a
freemason and served as
grand master of the
Grand Lodge of Pennsylvania from 1841 to 1842. ==Later life==