Congress Hall was built in 1816. It was first used as a wooden boarding house for guests to the new seaside resort of Cape May. The house's proprietor, Thomas H. Hughes, called it "The Big House" but many of the locals believed that it was too big to ever be successful. They nicknamed it "Tommy's Folly." In 1828, Hughes was elected to the US House of Representatives, and he changed the name of the hotel to Congress Hall.

A number of Presidents of the United States chose Congress Hall as their favorite vacation spot, including Franklin Pierce, James Buchanan, Ulysses S. Grant and Chester Alan Arthur. Benjamin Harrison initially vacationed there, and liked it so much that he made Congress Hall his official Summer White House. For the summer months during Harrison's Presidency, Congress Hall became the center of state business each year.
At the time, Philadelphia department store mogul John Wanamaker was developing Sea Grove at Cape May Point. Wanamaker became a major financial backer of Harrison's presidential campaign in 1888. Following his election, Harrison then appointed Wanamaker Postmaster General. It was Wanamaker who introduced Harrison to the resort area. The Wanamakers had a large Italian style cottage at Cape May Point called Lilenmyn at The Point. They invited the Harrisons as house guests durring the President's first summer in the White House in 1889. It is said that Harrison loved the fresh seafood, especially the oysters. First Lady Caroline Harrison said that if it were up to her husband, he would eat oysters three times a day. The First Lady also loved Cape May Point. She was an artist and especially enjoyed the ocean views and gardens.
Wanamaker decided to build the Harrisons their very own large cottage to ensure the President would return every summer. Wanamaked believed that Harrison's presence would generate publicity for the local tourism industry as well as investors in the local economy. A 23-room villa was built at the cost of $10,000 at Beach and Harvard Avenues. The President told Wanamaker he could not accept such an ostentatious gift, that it would be unethical. Wanamaker decided to gift the home to the First Lady in a White House ceremony in June, 1890. The large cottage became a scandal in Washington, as Wanamaker and his business partners were accused of buying presidential favor for railroad and housing developers. In spite of the controversy, the Harrisons continued to summer at Cape May Point.
To alleviate the controversy Harrison sent a checque in the amount of $10,000 to Wanamaker. Mrs. Harrison and her extended family and settled in at her brand new cottage on the beach. She frequently requested local coachman William Turner to take her for carriage rides around the area.
In 1891 Harrison made headlines internationally when he chose the first floor of Congress Hall as his Summer White House. A story in the New York Times reported on July 4th, 1891 read as follows:
President Harrison’s Fourth was scarcely a holiday. In the morning he walked up the beach with his grandchildren. Postmaster General Wanamaker reached Cape May Point at 11:30 o’clock on the express from Philadelphia. He called early upon the President and spent the afternoon with him in earnest work upon Post Office matters. After business had been disposed of Mr. Harrison took another long walk with Mrs. Dimmick. The President’s family had some fireworks during the evening and were further entertained by a similar display from atop the Cape May Point Lighthouse.
Mrs. Harrison's health became poor during her last summer at Cape May Point. She died one year later of tuberculosis at age 60. Benjamin Harrison sold the Cape May cottage back to John Wanamaker for $10,000 in 1896. The Wanamaker family used the President’s seaside retreat for several years and eventually turned it over to the Society for the Prevention of Cruelty to Children. The Wanamaker cottage, originally built at Beach and Harvard avenues, is now the Marianist Retreat and was moved inland. It still stands at Cape and Yale, looking much the way it did when President Harrison rocked on its porch with John Wanamaker.

No sitting President has visited Congress Hall since 1891. Today, Congress Hall is a fully functioning, high-end resort hotel.