Allen had married Harriet C. Dean of
Manchester, New Hampshire, in 1870, and they lived in Lowell on Rolfe Street, at their home called "
The Terraces", today part of UMass Lowell. They raised two daughters, Bertha and Louise. While Allen and his family were living in Puerto Rico when he was governor, his daughter Bertha Allen married naval officer George W. Logan in 1900. Their wedding was at the Palace, the governor's residence. The second daughter, Louise Allen (1875–1953), became a sculptor and a member of many artistic societies. Her son,
Allen Hobbs, was a hydrographer in the US Navy. He was appointed as the 32nd
Governor of American Samoa when it was a military position. Allen pursued a variety of leisure interests. He was an avid
painter, and completed twenty-seven landscape and marine paintings. These are held in the Allen Collection of the
Whistler House Museum of Art in Lowell. Interested in gardens, he ensured that the grounds of his home, "The Terraces," were planted with gardens, fountains, a pergola, and a large gazebo. The latter can be seen in photographs of the Allen House Collection,
University of Massachusetts Center for Lowell History. It was donated by Walter E. Hayes, his groundskeeper. Charles Herbert Allen died in Lowell and is buried in
Lowell Cemetery. A memorial was created to honor his name. ==Legacy and honors==