In the summer between his junior and senior college years, he competed in the
1976 Montreal Olympic Games winning golds in the 100-meter freestyle, the 4x200 meter freestyle relay, and the 4x100 meter medley relay, with each an Olympic and World record time. He also won a bronze in the 200-meter freestyle. Jim was the world record holder in the 100-meter event prior to the Olympic Games with a time of 50.59. Only 20 days after Jim set the new world record in the 100-meter freestyle of 49.99 during the '76 Montreal Olympics, University of Alabama swimmer Jonty Skinner broke Jim's record at Philadelphia's United States Summer National Swimming Championships by 0.55 seconds. At the Montreal Olympics, Jim was coached both by his Indiana Coach Doc Counsilman, who was Olympic Head Coach, and Assistant Olympic Coach
Don Gambril who was assigned to work with Montgomery by Counsilman. During his coaching career he has taken time to compete in organized open-water swims which include the Maui Channel Swim, the La Jolla Roughwater, the Alcatraz, the Chesapeake Bay swim, and France's Race across St. Tropez. He has had several age-group wins in the 2.4 mile Waikiki Roughwater swim. ==Coaching==