In 1852, Ayers sold his hotel and purchased , in Rochester, which he farmed for about five years. He became a stockholder in the Fox River Valley Railroad and moved back to Burlington to help run it for about a year. Ayers moved to a new farm in Rochester of , where he lived for the rest of his life. He continued to participate in politics, becoming a member of the Democratic Party and holding various local offices. In 1872, he was part of a group of men, including
J. I. Case, who founded the Bank of Burlington (which was eventually acquired by
Marshall & Ilsley Bank). At the time of his death, he was vice president of the First National Bank of Burlington. As of 1879, he was one of the owners of the
malt house on the east bank of the Fox River in Burlington, which was converted in 1947 into the Malt House Theatre. Ayers also owned a controlling interest in Burlington's
flour mill and considerable
real estate investments. In March 1880, Ayers gave a lot at the corner of Pine and State streets to the local
Episcopal congregation on the condition that a church be built on that lot, failing which the property would revert to the Ayers family. A cornerstone saying "St. John's" was laid in November 1880, but the church was never built, and ownership reverted to the Ayers family. After his death in June 1884, his family members inherited the land. ==Death==