Lydia Byam was born to parents William Byam and Martha Rogers (daughter of Edward Rogers). Through her father she was a great granddaughter of
Edward Byam (c. 1664-1741), Governor of the
Leeward Islands in 1715 and
Lieutenant Governor of
Antigua from 1715 until his death in 1741. She was related to William Gunthorpe, another Governor of Antigua and her extended family were woven through the rich white slaving owning class of Antigua. Surviving letters indicate that Byam is likely to have been educated in Britain before returning to Antigua. In 1835, Lydia and her daughter received £1,706 () in slave compensation for the Willis Freeman estate following the
abolishment of slavery in the British Empire. The prominence of the Byam family on the island is clear from the will her father wrote, describing the extensive estate and slaves he owned. William Byam's will, dated March 1773, left his estate in Antigua, real and personal, to the first son Edward Byam, 4,000 pounds () to son Samuel Byam, and 3,000 pounds () to daughter Lydia Byam; his wife Martha was granted the house and lands in Pembrokeshire, which was to be granted to Lydia at her death. == Personal life ==