Anderson started the Diamond A Orchard on his parents' farm in 1911 and returned to Shoreham permanently in 1920. There he became a prosperous farmer who cultivated an orchard totaling 3,500 apple trees, mainly
MacIntosh and
Northern Spy cultivars, by the early 1930s. At its zenith in 1943, 40,000 bushels of apples were harvested from Anderson's orchard. He served as president of the Vermont Horticultural Society in 1935 and experimented with pomology in collaboration with Governor
George Aiken. He spoke at Quebec Horticultural Society meetings in
Montreal in 1939, 1941, 1942, 1943, and 1944, delivering his speeches first in English and then in French, a language he had learned from his mother. In addition to his farming, Anderson served as chaplain and secretary of the state
Masonic society, whose lodges adopted a statewide policy of racial integration through a unanimous vote. Anderson served in a number of elected town offices, serving as a Shoreman school board member in 1921–1923, town auditor in 1926, and town agent from 1937–1946. He was elected to represent Shoreham in the Vermont House of Representatives in 1945 and served two terms (1946–1949). He won his first election by a vote of 218 to 4 and his re-election by 92 to 1. He was the second African American to serve in the state legislature since Alexander Twilight's election in 1836. In
Montpelier, the state capital, the Pavilion Hotel and the Tavern refused to lodge him on account of his race, so he lodged at Miller's Inn while the assembly was in session, befriending Middlebury College president
John Martin Thomas and other legislators. The newspapers generally praised him, with a 1945
Burlington Free Press editorial calling him "typical of the state without regard to his race or the color of his skin." Anderson corresponded and met with U.S. Senator
Warren Austin, a fellow apple enthusiast to whom Anderson had provided cuttings from the Diamond A. Orchard, to combat racial discrimination in the military. Secretary of War
Henry L. Stimson assured Senator Austin that "Mr. Anderson's suggestion has been given very careful study," pointing to the establishment of a
ROTC program at
Tuskegee University to increase the number of Black officers. == Later life and family ==