Hartley was born in 1849, second son of John Hartley (died 1884) and his wife Emma, daughter of ironmaster
George Benjamin Thorneycroft of
Wolverhampton. His parents were both of south Staffordshire industrial business families, his father's family owned the glass making firm of Hartley Chance & Company of
Smethwick, while the Thorneycrofts founded Shrubbery Ironworks in Wolverhampton, in which his father became a partner. He grew up at
Tong Castle in Shropshire, which his father leased from the
Earl of Bradford in 1856. In 1867, Hartley matriculated at
Christ Church, Oxford. He graduated B.A. in 1870, and M.A. in 1874. Hartley married Alice Margaret Lascelles Murray, daughter of
William Murray, 4th Earl of Mansfield and a granddaughter of
Henry Lascelles, 3rd Earl of Harewood, in 1875. They had no children. ==Career outside sport==