Without seeing the actual treatment note, it is impossible to say what level the visit should have been billed at. The payer may be doing what you said, or, the note may lacking what is needed to achieve a 99204, which would not surprise me in the least. There is a saying..."if it wasn't documented, it wasn't done". Meaning the provider may well have done everything need to achieve a 99204, and spent enough time with the patient, but if the note does not reflect that, it will be down coded to what it DOES reflect.
Achieving a 99204 requires A LOT of documentation, and it has to be pertinent to the presenting issues, and stand on its own. I audit treatment notes all the time, and I constantly see 99204's and 99205's(the highest new patient level) being billed, and the notes are only good enough for a 99201 or 99202. If you can get a redacted copy of the note, with all PHI removed, I can tell you more, but without it, it is anyone's guess to whether the payer is right or wrong here.