Joseph married Grace Adams (1789–1870) of
Swedesboro, New Jersey, on April 28, 1811. They had a daughter, Anna Maria, who married Isaac A. Pennypacker in 1839. Anna and Isaac's eldest son was
Samuel Whitaker Pennypacker; a judge, historian, and the first 20th century
Governor of Pennsylvania. When Isaac died in 1856, Anna and her children, including Samuel, came back to live in
Mont Clare with Joseph. Whitaker died of a heart attack on November 30, 1870, at his Mont Clare home. ==References==