The 4-story mansion was built on a plantation south of Asheville approximately twenty years before the Civil War. Constructed on a hill’s summit, ringed by picturesque mountains, the manse was constructed by slave labor. During a time when most people lived in log cabins, the imposing structure was composed of rare brick. Slave labor was probably used to construct the impressive home. Today known as The Smith-McDowell House, it is the oldest surviving house in Asheville and the oldest brick house in Buncombe County, North Carolina.

In 1974 The the house and grounds were purchased by Asheville-Buncombe Technical Community College in 1974. That same year, the Western North Carolina Historical Association rescued the house from demolition by negotiating a lease to restore the house as a heritage center. Due to fund-raising efforts and extensive restorations, the Museum opened in 1981. Today, the restored Smith-McDowell House is a nonprofit museum and is included in the National Register of Historic Places. It is the finest surviving example of brick antebellum architecture in Western North Carolina.