It is said that Madison acknowledged slavery to be a great evil, but he continued to own slaves himself and regard his enslaved laborers as his "property". After leaving the Presidency, he wrote to Francis Wright in 1825 using the word "evil" to describe slavery, despite his participating in the institution. He wrote:
The magnitude of this evil among us is so deeply felt, and so universally acknowledged, that no merit could be greater than that of devising a satisfactory remedy for it.
But Madison would not be the one to find that remedy.
Slaveholding was a tradition in Madison's family. His grandfather Ambrose Madison had owned a plantation called "Mount Pleasant." The house that Madison lived in, which came to be known as Montpelier, was built in the 1760s by slave labor. Ambrose died in 1732 from poisoning. Three slaves were accused of conspiring to murder him. They were tried and convicted of murder. James Madison, Sr., inherited the estate. According to his inventory, James Madison, Sr., owned 108 slaves in Orange County at the time of his death in 1801.
During the revolutionary war, Madison was asked to consider a proposal to permit enslaved men to serve as soldiers in the army. He suggested enlisting these persons and allowing them to trade their military service for their freedom. He wrote: "Would it not be as well to liberate and make soldiers at once of the blacks themselves as to make them instruments for enlisting white Soldiers? It would certainly be more consonant to the principles of liberty which ought never to be loss sight of in a contest for liberty." He continued to make statements acknowledging the contradiction between the institution of slavery and the cause of freedom At the Philadelphia Convention, Madison wrote "Where slavery exists the republican Theory becomes still more fallacious." While not in favor of ending the practice of slavery, he did call for an end to the importation of enslaved persons, though his final document barred Congress from interfering with the international slave trade until 1808. The domestic trade in slaves was expressly permitted by the constitution. Even then, Madison was opposed to the 20-year ban on ending the international slave trade. He wrote:
"Twenty years will produce all the mischief that can be apprehended from the liberty to import slaves. So long a term will be more dishonorable to the National character than to say nothing about it in the Constitution."
Madison ultimately accepted this as a necessary compromise to get the South to ratify the constitution. It was Madison who also proposed that apportionment in the United States House of Representatives be allocated by the sum of each state's free population and slave population, and this eventually led to the adoption of the Three-Fifths Compromise. (where 60% of a state's population of enslaved persons would be counted in calculating the number of seats that state was allowed in the House of Representatives.)
In 1808 when Madison was elected President, the world-wide demand for cotton had not yet peaked, and slavery was not yet as divisive an issue as it would come to be. Madison had considered the issue of slavery when he was one of the drafters of the Constitution. He approached the issue pragmatically and recognized its potential to derail his effort to reach consensus. Madison continued to acknowledge that slavery was a great evil, but he still continued to regard his slaves as his property. Whether his expressed opposition to slavery was his true sentiment or a means of appearing sympathetic to the views of those in the north opposed to slavery as a political tactic is unclear.
Madison owned slaves at his residence of Montpelier and his mother, Nelly Madison, and grandmother, Frances Madison, also owned slaves there. The 1820 Census notes the Madison household as having 15 free white persons, 106 enslaved African Americans (54 of which were men and 52 of which were women), and 0 free African Americans. The number of slaves he owned fluctuated. Madison's personal property tax records show that the least number of slaves for whom Madison was taxed was nine, and the most was 66.
After his Presidency, Madison supported the extension of slavery into the West during the Missouri crisis of 1819–1821. He also said that he believed that former slaves were unlikely to successfully integrate into Southern society. As early as the late 1780s, he became interested in the idea of African-Americans establishing colonies in Africa and he later became president of the American Colonization Society, which founded the settlement of Liberia for former slaves. His successor James Monroe was also involved in the Society and Monrovia, the capital city of Liberia, is named for Monroe.
Some of the slaves held by Madison were sold for financial reasons prior to his death in 1836. His wife Dolley also sold some after his death, and some were included in the sale of Montpelier to Henry Moncure in 1844. Dolley also willed some of the slaves to her relatives. Despite all of his rhetoric about freedom, none of his enslaved persons were given their freedom in his will. While Madison wrote that he had considered such an action, he never did so. Madison wrote that he believed that enslaved persons could not be freed unless "they are permanently removed beyond the region occupied by, or allotted to a white population." He continued to support the American Colonization Society's efforts to return freed former slaves to Africa, after indemnifying the slave owners. In his will he left $2,000 in trust to the Society, along with the proceeds from the sale of his grist mill.
Enslaved persons cared for Madison while he was sick. One of them who had been a longtime faithful servant, Paul Jennings, was at his side when he died. Despite this and despite all of his expressed reservations about slavery, and his presenting himself as a champion of freedom, Madison did not free any of his slaves in his will as George Washington had done.