Wylie is the son of Craig Wylie (1908–1976), one-time editor-in-chief at
Houghton Mifflin, and Angela Fowler (1915–1989), daughter of the landscape architect and artist Robert Ludlow Fowler, Jr, of Oatlands, New York (son of judge Robert Ludlow Fowler, author of many legal texts). Wylie grew up in
Sudbury, Massachusetts, and attended
St. Paul's School in
Concord, New Hampshire, from which he was dismissed in 1965; an interview with his university alumni magazine stated that this was for arranging illicit excursions to
Boston for fellow students and supplying them, illegally, with alcohol. When he was a teenager, he spent nine months in Manhattan's Payne Whitney clinic, a psychiatric hospital, for punching a police officer. He graduated from
Harvard. ==Family==