‘It is up to all of us to fix this,’ former president said in speech at Hamilton College in New York