Built as a commercial venture in 1793 by Elisha Wheeler and Asher Goodnow, the Federal-style Hosmer House is named in memory of Reverend Edwin Barrett Hosmer, father of artist Florence Ames Hosmer (1880–1978), who purchased the house in 1897.

A graduate of the Massachusetts College of Art, where she later served on the faculty, Florence Hosmer fully supported herself by selling paintings commissioned by dignitaries, musicians, friends, and relatives, and by teaching art in private and public schools. She maintained a studio and tearoom on Newbury Street in Boston in the 1940s and spent her summers at artist colonies in Peterborough, New Hampshire, Ogunquit, Maine, and on Cape Cod in Massachusetts. Hosmer’s paintings have been displayed by The Copley Society and The International Institute of Boston, and are in the permanent collections of the Essex Institute of Salem, Massachusetts, Dartmouth College in New Hampshire, the Danforth Museum in Framingham, Massachusetts, the Park Street Church in Boston, and in private collections.

Hosmer deeded the property to the Town of Sudbury in 1959 with the condition that the Town provide for her care until her death, stipulating the house and its contents remain on display to the public as a memorial to her father. She also left over 450 of her paintings to the Town, many of which are on display at the house.

The Hosmer House is managed by the Sudbury Historical Commission, and is open to the public on major holidays and the first weekend in December.