So What I infer from what you are saying is that I could perceive a wavelength of ~ 565–590 nm (yellow) as some other color taking for granted that I perceive color in a normal manner and am not color blind or suffer from a different perceptual defect. It could be yellow today and blue tomorrow, green and red the day after? While it is true that how we perceive the wavelength may differ from person to person the terminology refers to the particular band however it actually appears to us.
If color has no reality outside of our brains why are cardinals red and blue jays blue? Why are roses red and violets blue?
How we perceive the effect is immaterial. It is the effect that is being described. Ultimately everything we can describe is a result of how we perceive it, which is what I think you are really saying, but that is a matter of linguistics not essence. I we were able to perceive a different spectrum we would have different words.