Prosser was married three times and in 1877 was sued by Anna A. Hickey, a woman who claimed he proposed but refused to marry her. In 1834, he married Lucy Wilbur, and they had four children, including: • Ms. Prosser, who married John Armstrong of New York. •
Anna Weed Prosser (1846–1902), who did not marry but adopted three children; After the death of his first wife, he married Kate Muldary, a native of
Brooklyn, New York. They lived at 786 Delaware Avenue in Buffalo. In 1880, Kate committed suicide by jumping from the roof of a five-story tenement at 430 East 14th Street in New York City. He married, thirdly, to Mary A. Finnegan of Buffalo. He was buried at the
Forest Lawn Cemetery, Buffalo.
Descendants Through his son Henry, he was a grandfather of the banker and philanthropist
Seward Prosser (1871–1942), who served as the head of
Bankers Trust. ==References==