Dollardstown Castle was built by the D’Ullard brothers shortly after the Norman conquest in the late 12th century. They received permission to erect the castle and fortifications from Hugh DeLacy, the lord of Kilkea Castle. The castle was essentially a quadrangle with an inner courtyard and a surrounding motte.

One of the earliest references to Dollardstown is in the Aphorismical Discovery of Treasonable Faction, an account of the Cromwellian re-conquest of Ireland after the English Civil War. In the 1640s, the then occupant, Sir Hugh Hunt mortgaged the property to finance a troop of horse to fight for Charles I. The castle was later captured by Captain Gerald Fitzgerald operating under the command of Lord Castlehaven.

A sketch from the 1930s of Dollardstown House, Athy, Co. Kildare - Part of 'Ireland's Ancient East'Although the castle was razed, the townland’s name lived on with Dollardstown House, at the time a late Elizabethan dwelling, which is now the back of the present house. The house was sold in the 1690s to General Sir Richard Neville who, after campaigning successfully with the Duke of Marlborough, retired here as GOC for Ireland under Queen Anne. It is believed that he built the front of the present house shortly thereafter and planted a great number of trees, many of which still survive.

During the 1798 Rebellion the house was ransacked by the outraged Loyalist locals because the mistress of the house was a cousin of Lord Edward Fitzgerald, the leader of the rebellion. Despite being heavily pregnant at the time, she escaped through the back of the house and attempted to make her way across the fields to Kilkea Castle. Before she could reach safety, she gave birth to a son who later during his clerical career served as papal secretary to Pope Pius IX.