Elder was born in
Lebanon, Kentucky, in March 1921, the youngest of four children of George Rezin Elder Sr and Mary Ellen "Lilly" Elder. His mother died when George Jr. was one; the family moved to
Louisville, Kentucky, in the 1920s, then a few years later to Chicago and finally to Los Angeles. He served as a Marine artilleryman in the Pacific theatre, including in the
Battle of Iwo Jima. Elder was a
running back for the
Fordham University football team. He considered attending the
University of Notre Dame, where his brother Jack had been a running back. Elder, who ran the 100-yard dash in 9.9 seconds, said that he chose Fordham to avoid comparisons to his brother. Elder ended up playing for the freshman team in 1941 and the varsity team in 1942. In October 1942, he sustained a broken foot in a game against
West Virginia University. In the summer of 1943, Fordham shut down its football program. Elder went to
Dartmouth College that fall as part of the
V-12 Navy College Training Program. When he arrived at Dartmouth, he was described as dealing with some chronic injuries, and he elected to play baseball instead of football. After serving in the
Marine Corps in
World War II and seeing action at
Iwo Jima, Elder was discharged and finished college at the
University of California, Los Angeles, ==Baseball career==