Do extracurricular activities matter much for admissions? I'm not sure whether I should be worried about them or not. I thought that most colleges did take them into account, but I've heard some conflicting things.
Extracurriculars matter to varying degrees depending on how rare and impressive they are and how competitive the school is. Someone who has slightly lower than average stats might be admitted to a school because of a unique extracurricular activity that demonstrates strong leadership potential (something like starting their own successful charity organization or winning a national competition). This same person probably wouldn't be admitted, however, if they had relatively run-of-the-mill extracurricular activities (played a sport, were in NHS) or none at all.
In general, extracurriculars also gain importance in the admissions process as schools get more selective. At the top schools in the country, all serious applicants will have exceptionally high grades and test scores, so the only way to stand out is through extracurricular activities. If you meet the baseline academic expectations of these schools, the strength of your extracurricular accomplishments may determine whether or not they decide to accept you.
Essentially, there are two scenarios where you should be concerned about extracurriculars strongly affecting your admissions chances: You're hoping to attend a school that is a reach based on your test scores and grades and/or you're applying to the most competitive colleges in the country.