He married Sarah Rodman (1793-1888), She was born on October 31, 1793, and had lived in
New Bedford, Massachusetts. They had five children between 1821 and 1836: Emily, Samuel, Isabel, Elizabeth and Clara. Between 1820 and 1840, Sarah and Charles employed 45 women, for several months each, who may have lived with them. A known employee was
Polly Johnson, who with her husband Nathan took in
Frederick Douglass when they had their own house. Peter Emanuel postulates that this was to give former slaves a "foothold" to establishing a successful life in New Bedford. Morgan wrote in 1841, "Sooner or later the minds of all good men must come to his conclusions & act upon them, be the consequences what they may." In a tribute following his death, his family and community were said to have lost a "kind and indulgent father, his wife an affectionate husband, and the community in which he resided a benevolent, enterprising, honest, christian man..." ==Career==