Withers wrote his autobiography,
Memoirs of an Octogenarian during his retirement. His extensive collection of personal papers are housed with the Kegley Library of
Wytheville Community College. Withers died at the "Ingleside"
plantation in Wytheville on September 21, 1907. He was buried in the East End Cemetery in Wytheville. In 1846 Withers married Mary Virginia Royall, with whom he had eight children, one of whom, Betty Ellison Withers Putney, was an advocate for
women's suffrage in the early 20th Century. Withers' grand nephew
Withers A. Burress was a commandant of
Virginia Military Institute and served as a combat commander in both
World War I and
World War II. He was also a cousin to
Thomas W. Chinn, Member of the
United States House of Representatives from
Louisiana and the US Minister to the
Kingdom of the Two Sicilies. Withers was a distant relative of figures such as
George Washington and
Robert E. Lee, as well as a direct descendant of
Nicolas Martiau, founder of
Yorktown, Virginia. ==References==