Withers was the second son of William Withers, a linen-draper, of St. Mary-le-Bow, London, and his first wife Sarah Cornish, daughter of George Cornish. He was a descendant of Sir Thomas Wyther and was believed to be the first family member to spell the name Wyther as Withers. It is sometimes difficult to distinguish father and son in the record, and Withers may have been the common counsellor for Cheap from 1675 to 1680. In 1680, like his father, he became a freeman of the
Fishmonger's Company. He married Margaret Hayes, daughter of Thomas Hayes of
Chertsey Abbey,
Surrey by licence dated 24 February 1682. ==Career==