The Justus Ramsey House, a 170-year-old structure that's recorded as St. Paul’s oldest private residence, could be demolished or moved from its current site on a restaurant patio.