Castle was born on August 29, 1929, in
Jersey City, New Jersey. He played as a football
quarterback at
St. Lawrence University, where he received his
bachelor's degree. He earned a second degree from the
Berkeley Divinity School in
New Haven, Connecticut. While still studying at Berkeley, was given an assignment to serve at a predominantly
African American Episcopal parish on
Manhattan's
Lower East Side, sparking a lifelong commitment to civil rights and minority rights. His first assignment as an Episcopal priest was as a rector at
St. John’s Episcopal Church in his native Jersey City from 1960 to 1968. During that time, Castle became very involved with the
American civil rights movement, including traveling to
Mississippi to march with
Martin Luther King Jr. He became one of the city's most vocal activists. He once dumped large amounts of garbage outside Jersey City Hall as a way to draw attention to the need for more street cleanings. He protested against his own New Jersey Episcopal bishop for his membership in
segregated social clubs. Castle also led protests against restaurants, banks, and other businesses because they would not hire minority employees. An opponent of the
Vietnam War, he allowed 1960s left-wing groups, including the
Black Panthers and
Students for a Democratic Society, to use both his home and St. John's Episcopal Church for their meetings. His views and actions in Jersey City proved so controversial that when he left his post St. John's Church in 1968, no other church in the
Episcopal Diocese of Newark would accept him into their parishes. Castle and his family moved to
Vermont, where he became involved with social work and operated a
general store. Castle was also the football coach for several years at
North Country Union High School where he led the team, with his son John at quarterback, to the state championship game in 1981. ==Rector at St. Mary's Episcopal Church==