Madigan's father Kendall Morgan Madigan (1908–1938) was a doctor and mother Elsie Maud Loydstrom (1911–1983) was a dentist. Madigan attended
Waverley College in Sydney where he took boxing lessons with Australian champion Hughie Dwyer and sparred with leading professional boxers. In the 1950s, he spent time in the United States being coached by leading trainer
Cus D'Amato. After returning to Australia, he sold
EH Holdens with rugby league player
Rex Mossop. On 17 January 1955, Madigan suffered serious injuries in a car crash in
Bavaria,
West Germany. His 23-year-old passenger Helen Stokes-Smith was killed when on an icy road Madigan lost control when trying to avoid a parked truck. Madigan married a German psychotherapist, Sybille, in November 1960 and their son Kendall Morgan Madigan was born in August 1961. Madigan is also father to a daughter, Elizabeth Phy Collins who was born in NYC in 1960. In the mid-1960s, Madigan sold property investments and had a successful modeling career in London. He then moved to New York City and commenced modeling with
Howard Zieff, a renowned photographer. ==Rugby union==