In 1866, Stumm became a member of the Methodist church. At age 17, he started teaching Sunday school near
Paradise in Muhlenberg County. Under the advice of Rev. Samuel Elliott, he joined the
A.M.E. Church in
Hartford in Ohio County, Kentucky and became a licensed minister. By age 19, he became a teacher, which was unusual during that era. While he was still attending university, Stumm was asked by a local newspaper editor to report on the proceeding on the convention of the Baptists. As a result, he acted as a correspondent for various newspapers including
The Standard (Paducah, Kentucky newspaper),
The Pilot (Nashville, Tennessee newspaper),
American Baptist (Louisville, Kentucky newspaper),
The Tribune (Danville, Kentucky), and the
Baptist Companion. He wrote a children's column called "Uncle Charles" for the
American Baptist. Stumm also had a column in the
Bowling Green Democrat. He published the
Bowling Green Watchman for many years, where his wife served as a journalist. At age 22 around 1870, Stumm built his first church building in Chaplaintown, Kentucky. He had witnessed a religious debate between Methodist and "
Campbellite" preachers, which prompted him to look deeper into the Baptist religion and church leadership. He eventually converted to Baptist, joining the First Baptist Church of Nashville, Tennessee led by Rev. Nelson G. Merry. Within the year of joining the new church, he was licensed as a minister and a year later he was
ordained. He married Elizabeth Penmen in 1875, a fellow student from Berea. Soon after their marriage, Stumm was called to missionary work to run a church and spent two years at his first station, before being transferred to
Elizabethtown, Kentucky. From 1879 to 1881, Stumm was placed as minister of the Independent Baptist Church of Frankfort. This was followed by work at the largest church in the state in Bowling Green, and later work as assistant pastor at the Spruce Street Baptist Church in Nashville, Tennessee. Stumm was briefly the pastor of the Ebenezer Baptist Church in Boston, Massachusetts but left due to a dislike of the weather. In October 1885, he joined the struggling Union Baptist Church in Philadelphia, where he increased the size of the congregation and was able to build a new church building. He was an active participant in the Baptist Ministers' Conference, which was attended by both Black and White clergy. In 1887, he became the editor of the
Baptist Monitor, a newspaper by the New England Missionary Convention. In 1890, the Stumms began publishing
The Christian Banner, a religious journal, for which he acted as the editor, and his wife served as the business manager. He was awarded an honorary
doctorate of divinity in 1890 from the
University of Louisville. == Late life and death ==