, ca 1776 Lord Derby married
Lady Elizabeth, daughter of
James Hamilton, 6th Duke of Hamilton, on 23 June 1774. They had three children: •
Edward Smith-Stanley, 13th Earl of Derby (21 April 1775 – 30 June 1851); married his cousin Charlotte Margaret Hornby, daughter of Reverend Geoffrey Hornby by his wife, the Hon. Lucy Hornby (née Smith-Stanley) • Lady Charlotte Stanley (17 October 1776 – 25 November 1805); married her cousin Edmund Hornby, Esq., son of Reverend Geoffrey Hornby by his wife, the Hon. Lucy Hornby (née Smith-Stanley) • Lady Elizabeth Henrietta Stanley (29 April 1778 – 4 November 1857); married Stephen Thomas Cole, Esq., of Stoke Lyne, Oxfordshire, and Twickenham, and had issue. In the late 1770s, Lady Derby had a very public affair with
John Frederick Sackville, 3rd Duke of Dorset. In 1779, the countess moved out of Lord Derby's house, leaving their children behind, apparently expecting that her husband would agree to a divorce and that the Duke would then marry her. About one year after she left his house, Lord Derby made it known that he had no intention of divorcing his wife; at the same time, he continued to deny her access to her children. The countess was socially ostracised for the remainder of her life. Historian Peter Thomson suggests that the third of the couple's children, Lady Elizabeth Henrietta, was the result of Lady Derby's affair with Dorset. Despite this, the Earl of Derby cared for the child after his wife left him. Lady Derby died at the age of 44 on 14 March 1797. Six weeks later, on 1 May 1797, Lord Derby married the actress
Elizabeth Farren, daughter of George Farren. They had three more children: • Lady Lucy Elizabeth Stanley (1 March 1799 – 25 April 1809), died young • Hon. James Stanley (9 March 1800 – 3 April 1817), died young "after a long and painful illness" • Lady Mary Margaret Stanley (23 March 1801 – December 1858); married
Thomas Egerton, 2nd Earl of Wilton. Lord Derby survived his second wife by five years and died on 21 October 1834, aged 82. He was succeeded in the earldom by his son from his first marriage,
Edward, Lord Stanley. File:Lady Elizabeth Hamilton (1753–1797), Countess of Derby.jpg|
Portrait of Elizabeth, Countess of Derby, by
George Romney, ca 1776-1778 File:Portrait of Elizabeth Farren, by Thomas Lawrence.jpg|
Portrait of Elizabeth Farren, by
Thomas Lawrence, 1790 File:Arms of the Earl of Derby 02666.jpg|Arms of Stanley, Earl of Derby: ''Argent, on a bend azure three buck's heads cabossed or''. Painted in 1781 ==Arms==