Dolley Payne was born on May 20, 1768,to Mary Coles and John Payne Jr. and lived with her family in a log cabin in New Garden, Guilford County, North Carolina| (present-day
Greensboro), North Carolina. Her parents had married in 1761, uniting two prominent
Virginian families. Little is known about the family's life before 1793, when Madison was 25, because few documents have survived; Madison's earliest known letter dates to 1783. Mary Coles was from a
Quaker family and two years after their marriage the couple applied for membership in the Cedar Creek
meeting. The application was considered for a very lengthy time before they were admitted in 1765. He would become a fervent member of the faith. The family had moved to New Garden, a Quaker community, in 1765. Madison was the family's third child and first daughter. The family had an enslaved
nursemaid. In early 1769, the Paynes returned to Virginia for reasons that are unclear. Her father did not participate in the
American Revolutionary War, as his faith practiced pacifism, and Allgor writes that Madison was seemingly little affected by it. By 1783 John Payne had
emancipated his enslaved people, Payne, as a Quaker, had long encouraged
manumission, in Philadelphia, with whom she became lifelong friends. During her early years, Payne likely received formal education, though it is not known what this was. Allgor concludes that it was likely better than most Americans at the time, while Côté notes that it was probably "no more than a basic" one. Madison grew into a young woman who Côté writes was described "as one of the fairest of the fair". Upon the family's move to Philadelphia, John had attempted to build a career as a starch manufacturer, but the business failed in 1789. This was seen as a "weakness" at his Quaker meetings, for which he was expelled. He was devastated by this failure and died on October 24, 1792. Mary Payne initially made ends meet by opening her home as a boardinghouse beginning in 1791. Before his death, John had arranged Madison's marriage to John Todd, a Philadelphia lawyer. According to Allgor, Madison had rejected marriage with Todd previously and John's marriage arrangement was "manipulation". Conversely, Côté considers their marriage to have been "for love, not just duty". They were married on January 7, 1790, at a Quaker meeting house. Madison's friend Eliza Collins was her
bridesmaid. The couple moved several blocks away into a high-quality neighborhood.
Marriage and family Madison and Todd had two sons,
John Payne (called Payne, born February 29, 1792) and William Temple (born July 4, 1793). According to Allgor, their marriage grew into a "a loving happy partnership." Madison's sister Anna Payne moved in with them. In August 1793, a
yellow fever epidemic broke out in Philadelphia, killing 5,019 people in four months. Madison was hit particularly hard, losing her husband, son William, mother-in-law, and father-in-law. Two of her older brothers died just two years later, and she "never fully recovered" from the emotional toll of these deaths. While undergoing the loss of much of her family, she also had to take care of her surviving son without financial support. Her husband had left her money in his will, but the executor, her brother-in-law, withheld the funds, and she sued him for what she was owed.
Aaron Burr, who had once stayed at the boarding house of Madison's mother, assisted her in these efforts, offering legal advice. In a will, written around that time, Burr was named the guardian of Madison's only surviving child. ==Second marriage (1794–1800)==