Comer was born in 1842 to John Fletcher Comer (1811–1858) and Catharine Lucinda Drewry (d. 1898). He had five brothers:
John Wallace Comer (1845–1919), St. George Legare Comer (1847–1870), U.S. senator and Alabama governor
Braxton Bragg Comer (1848–1919), John Fletcher Comer, Jr. (1854–1927) and Edward Trippe Comer (1856–1927). There was also at least one daughter in the family. Comer's father was a cotton planter who also owned a lumberyard and a corn mill. He married Catharine Drewry in 1841, but left her a widow with six children when he died in 1858. ==Career==