Elwes was baptized in July 1683, the eldest son of
Gervase Elwes of
Stoke College and his wife Isabella Hervey, daughter of
Sir Thomas Hervey of
Ickworth, Suffolk. His father died in about 1687. As a child, he suffered from consumption so that he had a poor constitution and a thin spare body. He was timid, and extremely diffident, had no friends and no interests apart from hoarding up his money and partridge setting. He was admitted at
Queens' College, Cambridge on 22 June 1702. His grandfather
Sir Gervase Elwes, 1st Baronet died on 11 April 1706 and he succeeded to the
baronetcy and estate. The estate was so debt-ridden that his uncle,
John Hervey, 1st Earl of Bristol, advised him either to sell his lands or marry a rich wife. In the event he never married nor sold the estates but lived a life of abject penury. ==Political career==