Pierce endorsed Buchanan, even though the two were not close. Buchanan was elected, but the Democratic percentage of the popular vote in the North fell from 49.8 percent in 1852 to 41.4 in 1856. Buchanan won only five of sixteen free states (Pierce had won fourteen). In his final message to Congress, delivered in December 1856, Pierce attacked Republicans and abolitionists.
After leaving the White House, the Pierces remained in Washington for three, staying at the home of former Secretary of State William Marcy. Buchanan replaced all of Pierce's appointees with his own. The Pierces moved to Portsmouth, New Hampshire. Pierce's wife Jane was quite ill, suffering from tuberculosis. After leaving Washington, the Pierce's traveled by train to Philadelphia on March 25th so that Jane could see a doctor that was recommended by James Campbell. Her health appeared to improve well enough that she was able to accompany her husband on horseback rides. On May 20th they took the train to New York, where they stayed at the home of former Whig Governor Hamilton Fish. In June they left for home. Jane was left at her sister's home in Andover, Massachusetts, while Pierce went on to New Hampshire. Jane joined her husband and the two of them lived in a hotel at Portsmouth for the summer.
Pierce decided that Jane needed to be somewhere warm for the winter months. In December of 1857 he managed to secure passage for the two on them on a US Navy vessel bound for the Portuguese island of Madeira. His friend Clarence March had an uncle who was the US Consul there and the uncle agreed to host the ex-president and his wife. The Pierces stayed in the spacious home of Howard March for the next six months. While they were there, Jane's health continued to improve. Within weeks, she was taking rides on one of the two horses that they had brought along with them. They also took French lessons. Franklin Pierce learned the language better than his wife. They did so for a tour of the content that they had planned.
In June of 1858, the Pierces left on a tour of Portugal, Spain, France and Switzerland. In the fall they traveled to Florence, Naples, and Capri in Italy. In February of 1859 they arrived in Rome, where Pierce had a reunion with his old friend Nathaniel Hawthorne. The two had not seen one another for nearly six years. Hawthorne had been living in Rome since the fall of 1858. Hawthorne and his wife Sophia had their problems. Their teenage daughter Uma was unwell, and the Pierce's gave support to their friends while she recovered from her illness.
Following Una's recovery, the Pierces resumed their travels, going to Venice, and then to Vienna and into Germany. Jane's health began to get worse and she saw several doctors. From here the Pierces made their way back home, traveling to Brussels, Paris and then to London. They sailed for the United States in August of 1859 and arrived in Norfolk, Virginia in early September. By September 11, 1859 Jane was back in Andover.
Jane's continuing poor health called for a warmer climate and in January of 1860 the Pierce's traveled to Nassau in the Bahamas where they remained until the middle of May.
Pierce never lost sight of politics during his travels. His friends Jefferson Davis (who had served as Secretary of War in Pierce's cabinet) and James Campbell (who had been Postmaster-General in his cabinet) had urged Pierce to seek the Democratic Party's nomination for President in 1860, but Pierce declined to do so. When the war broke out in 1861, Pierce publicly opposed President Lincoln's order suspending the writ of habeas corpus. He said that even in a time of war, the country should not abandon its protection of civil liberties. This stand won him admirers with the emerging Northern Peace Democrats, but his political opponents saw the as further evidence of Pierce's southern bias. He was accused of conspiring with Confederates during the war, leading to a confrontation with Secretary of State William H. Seward. Pierce spoke out against the institution of the draft and the arrest of outspoken anti-administration Democrat Clement Vallandigham. He gave an address to New Hampshire Democrats in July 1863 that was highly critical of Lincoln. Pierce's reputation in the North was further damaged the following month when the Mississippi plantation of the Confederate president, Jefferson Davis, was seized by Union soldiers. Pierce's correspondence with Davis, all pre-war, was discovered, revealed his deep friendship with Davis.
In December of 1863, Jane Pierce died of tuberculosis in Andover, Massachusetts. She was buried at Old North Cemetery in Concord, New Hampshire. Pierce was further saddened by the death of his close friend Nathaniel Hawthorne in May 1864. He was with Hawthorne when the author died unexpectedly.
Some Democrats tried again to put Pierce's name up for consideration as the Democratic nominee for President in the 1864 presidential election, but Pierce was not interested in running. When news spread of Lincoln's assassination in April 1865, a mob gathered outside Pierce's home in Concord, demanding to know why he had not raised a flag as a public mourning gesture. Pierce met the crowd and expressed his sadness over Lincoln's death. He told them that his history of military and public service proved his patriotism, which was enough to quiet the crowd.
Pierce's drinking worsened in his later years. His health began to decline. He returned to Concord in September of 1869, suffering from severe cirrhosis of the liver. He died at 4:35 am on October 8, 1869. President Ulysses Grant, who later defended Pierce's service in the Mexican War in his memoirs, declared a day of national mourning. Pierce was interred next to his wife and two of his sons at Concord's Old North Cemetery.