
After he left office, Madison retired to Montpelier, his tobacco plantation in Orange County, Virginia, about 30 miles from Thomas Jefferson's home of Monticello. Madison was 65 years old and his wife Dolley was 49. It was her wish for the two of them to travel to Paris, but like his predecessors, Madison left the presidency a poorer man than when he entered it. He had left management of his plantation up to Dolley's son John Payne Todd who had mismanaged it. This and the low price of tobacco contributed to Madison's money problems.
Madison refused to allow his notes on the Constitutional Convention to be published in his lifetime. He believed that if they were published after his death, they could be sold by his estate in order to raise money for Dolley's support. Some of his biographers believe that Madison's financial troubles weighed heavily on him, and they contributed to his deteriorating mental and physical health.
In his later years, Madison became extremely concerned about his legacy and about how history would remember him. In an attempt to have himself cast in a better light, he began to modify his copies of letters that he had sent and other documents in his possession, to make himself appear more favorable. He changed days and dates in some of these, he added and deleted words and sentences, and even altered punctuation.
But the editing was more than just grammatical or punctuation. For example, he edited a letter written to Thomas Jefferson in which he criticized the Marquis de Lafayette. Madison not only inked out original passages, but imitated Jefferson's handwriting as well in making changes. He also edited out some of his criticisms of George Washington, and he altered the dates on letters he had sent that were critical of Alexander Hamilton. He tried to hide the fact that he and Thomas Jefferson had authored the Kentucky Resolutions, a series of political statements drafted in 1798 and 1799, in which the Kentucky and Virginia legislatures took the position that the federal Alien and Sedition Acts were unconstitutional and that the states had the right and the duty to refuse to follow acts of Congress which they believed were not authorized under the Constitution. The Kentucky Resolutions of 1798 stated that each individual state has the power to declare that federal laws are unconstitutional and void, that they could use nullification as a remedy. (These resolutions were later relied on to support nullification by John C. Calhoun and later to support secession by southern states at the beginning of the civil war.)
By the time he had reached his late seventies, this practice of editing his histrical records became an obsession with Madison. In 1829, at the age of 78, Madison was chosen as a representative to the constitutional convention in Richmond for the revising of the Virginia state constitution. Historian Drew R. McCoy, in The Last of the Fathers: James Madison and the Republican Legacy, wrote: "The Convention of 1829, we might say, pushed Madison steadily to the brink of self-delusion, if not despair. The dilemma of slavery undid him."

Madison sold some of his slaves to compensate for financial losses on his plantation. In his later years he found himself ignored by the new generation of political leaders. He died at Montpelier on June 28, 1836 at the age of 85.