James Sherman was born in Utica, New York on October 24, 1855. His parents were Richard Updike Sherman Mary Frances Sherman (described as being Richard's "distant cousin".) He was related to Roger Sherman, a signer of the Declaration of Independence, and famed Civil War General William Tecumseh Sherman. James Sherman was educated at Hamilton College, where he was said to be skilled in oratory and debate. He obtained his law degree and was admitted to the New York State bar in 1880, practising at the firm of Cookingham & Martin in Utica, and also serving as president of the Utica Trust & Deposit Co. and the New Hartford Canning Co. He also became mayor of Utica when he was twenty-nine. In 1881, he married Carrie Babcock of East Orange, New Jersey, and they had three sons.
In 1886, Sherman was elected U.S. Representative from New York's 23rd congressional district as a Republican, and he served twenty years in the House, with only a two-year interval (from 1891 to 1893). When the Republican party was divided over protective tariffs, Sherman sided with President William McKinley. He supported the gold standard against the potentially inflationary 'free silver'. Sherman never held a party leadership post or chaired a major committee until he appointed Chairman of the 'Committee of the Whole' - a committee that made decisions regarding the passage of bills.
In 1908, Sherman was nominated as the Republican candidate for Vice President on the ticket with William Howard Taft. Although he didn't have a high profile, he complimented Taft on the ticket by being an Easterner and a conservative. The Republicans won by a comfortable margin, though Sherman didn't receive much credit for the victory.
At first, Sherman and Taft disagreed over both tariff policy and the role of the Vice President. But Taft shifted politically from being a progressive to a conservative and the two of them worked together well as that progression took place. First Lady Helen Taft enjoyed the company of Sherman and his wife. Taft said of his Veep that Sherman accomplished much on Capitol Hill by his "charm of speech and manner, and his spirit of conciliation and compromise", backed by a "stubborn adherence" to his principles.
Taft split with his old friend and mentor ex-President Theodore Roosevelt, and when Roosevelt failed to win the Republican nomination for President in 1912, he walked out and formed his own Bull Moose party. This split the Republican vote, making re-election for Taft virtually impossible. Undaunted, Taft and Sherman campaigned on the same ticket in the 1912 contest, the first time a sitting Vice President had been re-nominated in eighty-four years.
Unfortunately, Sherman's health took a turn for the worse, due to his steadily worsening kidney condition (known as Bright's disease. Just days before the election, he died at home in Utica on October 30, 1912 at the age of 57. Taft was left with no running mate with less than a week before the November 5 election and the office of Vice-President remained vacant until March 4, 1913.