The difference between a 2320 and a 2400 is pretty small. As your guidance counselor said, most schools aren't going to accept or reject you on the basis of that score difference. And as your classmates said, there is a pretty high chance that you'd score lower on a retake. We have a handy tool right here that can help you figure out how much your chances of admission would change with an SAT score of 2320 vs a 2400 (or a 1570 vs 1600 on the new SAT). Even for the most competitive schools, it's not a huge difference, and you probably don't need to take the SAT again.
That said, if you're applying to a competitive college that superscores the SAT, it's possible that retaking the SAT could net you a higher superscore (although, as we discuss in this ExpertHub answer, it's unlikely that schools will superscore between the new and old SAT). But even then, the effort that you'd expend on getting those last 80 points could be better spent honing your personal statement.