Henry Farwell, son of William, lived here at the time of the Revolution. He was born in 1724, and was a soldier in the French and Indian Wars. He was captain of one of the companies of minutemen of Groton at the start of the Revolution, and marched to Cambridge with his company on the 19th of April, 1775. He was severely wounded at Bunker Hill by a musket-ball that passed through his body, lodging in the spine. It was extracted, and he engraved "1775" upon the ball and kept it as a precious relic until he died in 1803. At one time he was the owner of a large part of the land on which Ayer village stands, and resided in the vicinity of Phelps mill, where he owned and operated a saw and grist-mill. According to the 1890 History of Middlesex County, his dwelling house was quite "commodious." After he sold it to George Peirce in 1758, the place was used as a tavern. It is said that he then lived on the George Little place up through the time he marched to the Battle of Bunker Hill.   

Two main points of entry into Ayer converge at the Frederick Carlton Circle, a five-way junction that includes Barnum Road, Sandy Pond Road, East Main Street, Littleton Road, and Harvard Road. All these roads are designated as minor arterial/major collectors. The large heavily landscaped circle located east of the downtown area is an important gateway to the town. At the northwestern edge of the circle there is another triangular green space between the east and west bound travel lanes of East Main Street which makes the area feel somewhat like a parkway. Residents recognize the importance of maintaining this feel against the encroachments of fast food vendors.

MacPherson Road extends along the east side of the Nashua River between Bishop Road on the north and West Main Street on the south. It is presently a two-lane paved road parts of which are subject to periodic flooding. The land along MacPherson Road and the Nashua River is controlled by U.S. Fish and Wildlife as part of the Oxbow National Wildlife Refuge and has been identified as an area of critical environmental concern. (Squannassit ACEC). Tension exists between those who would like to see the road used to relieve Central Avenue/Main Street traffic congestion and those who are concerned about the possible degradation of an ecologically sensitive area.

Running east/west through the center of Ayer, Main Street is the primary civic and commercial district in Ayer. The downtown area burned in 1872 and most of the extant buildings date to the late 19th and early 20th centuries. The commercial district includes a variety of one to four-story building types, most built in masonry after the fire. The first building erected after the fire is the Page/Moore Block at 33-47 Main Street. A four-story Italianate building, it occupies a full block with an awning that extends out over the sidewalk, spanning the entire façade. The handsome Town Hall, a High Victorian Gothic structure built in 1873, was recently renovated and received a preservation award from the Massachusetts Historical Commission. Between the Town Hall and the 1989 Classical Revival bank building is a newly constructed mini-park.

Between 1971 and 1999 Ayer lost more than half of its agricultural land, leaving only 125 acres, most of which is hayed. The two remaining farms on the east side of the Ayer/Groton Road (North Washington Street) evoke the agricultural past of this part of town. Neither farm is extensively farmed any longer, but the views of the Eliades Barn in the midst of a rolling hay field along with the Smith Farm land buildings provide pleasant views. While some land is under permanent conservation restriction, proposed hospital expansion on the west side of the road opposite the Smith Farm would impact the character of the area.

This neighborhood lies immediately north of the town center and consists of a mix of late 19th and early 20th century wood frame houses in a range of styles. In general the houses along Washington Street are larger houses set on more spacious lots and almost all single family residences, while on Pleasant Street and the many side streets houses are generally smaller and include some multi-family dwellings. The neighborhood is valued because of the tree-lined streets, well-kept houses, visual variety and proximity to downtown.
The Pleasant Street School is listed on the National Register and the Historical Commission has been working to try to find an appropriate adaptive reuse.

In the 19th century, Ayer was an important railroad junction with three railroad lines traveling through town. In and around the rail line are a number of railroad artifacts of historical interest: a turntable, a signal tower, track switches, bridges and abandoned rail lines that all tell an important aspect of the town history, although today only the Fitchburg line of the Boston and Maine Railroad is active. Another issue of concern for townspeople is the recurring proposal to combine the Ayer and Shirley stops into one, since both towns rely on the proximity to the rail line for associated business and residential traffic.

The West Main Street neighborhood is a small enclave of residential properties on the north side of the street.  It includes a line of modest gable-front Greek Revival houses as well as two larger multi-family dwellings that have some Greek Revival detailing.