Why two, three or four types of cone cell? Why not, like, five thousand types of cone cell?

Where would you put them? Photoreceptor cells only detect light that lands on them. If you have, say, 3 types of cone, the next nearest of those cone cells can be very close by. If you have 5000, the next same cell is going to be far away. So if you're going to have any kind of spatial resolution to your color map you're going to have to average anyways and lose the spectral resolution you added.

What would you gain? You can already perceive many many colors with just 3 cone cells. Wavelength is one-dimensional. There are some combinations that can't be resolved but they are rarely produced by real objects.

You misunderstand how evolution works.

It is not about best it is about good enough and 3 colors is good enough for primates. Each variation needs to come from a mutation and be favored and the gain just is not there, for that many types of cone cells. A new color requires a new protein which is harder to get. That new protein then needs to make enough of a survival or mating advantage to spread.

Also you are confused about how hearing works, we don't have different types of hair cells to detect difference frequencies, we have the same hair cells located in different places along the length of a shaped spiral. Its the same cell cluster over and over again just in different locations. geography gives us frequency, for the eyes location gives us direction.

If you look at an image of a hair cell below, those are not separate cells but a single cell.

Keep in mind birds and reptiles have a base 4 color system, those 4 cells give a pretty even sampling of the light that makes it to the earths surface. Our ancestors (basal mammals) lost 2 of them possibly to get better night vision and then primates mutated to get a new 3rd. Being able to see reds again helped our ancestors identify ripe fruit a fairly large advantage for a group that eats a LOT of fruit. But it is janky solution, its not an even sampling so its imprecise. You carry the evolutionary baggage of your dichromatic ancestors, birds see colors you and I will never see.

In addition to the answer by @BryanKrause

There are a couple of good reasons why ears need many different sound receptors while eyes only need a few.

Ears, while they are a primary sense organ, are not our main sensory input for daily life. Echolocation or other sound reception doesn't add to our spatial resolution in the same manner that visual resolution does. Basically any sound that can be detected gives you some information about 1) the location and 2) they type of object it is coming from, so you don't need discriminatory function too much. Even reflected sound in the form of echolocation (yes, humans do this, often unconsciously) does this - it tells you where an object is, and that there is a solid object there and sometimes information about the density of the object too.

In contrast, the eyes have a much greater role in our life - colour discrimination is important as it tells us not only what an object is, but also information about the object - it's the difference between a ripe peach and an unripe one, or the difference between a poisonous berry type and an edible one something you can't easily tell from the sound alone. Those animals that rely on sound for obtaining food are usually predators, so don't need to rely on vision as to whether something is poisonous or not - and smell can help there too. For example they could tell the difference in smell a between frog (not poisonous) and a toad (poisonous).

In terms of the physics, sound detection is a bit harder than optical detection - you need to detect minute pressure changes (sound waves), which is performed by the ear using a resonance system - hair cells vibrate in sympathy with the sound vibration and this is done using the inner ear, which kind of funnels sounds in and has zones of resonance for different frequencies, so you need different neurons corresponding to those zones. Without the zonal system, you wouldn't have the physical space to have different hairs which are capable of detecting the different frequencies. Perhaps a different evolutionary path could have had detectors that measured how much a particular hair vibrated in response to sound, making it more analogous to a visual receptor, but I think it would be very complex mechanically to get it to work as each hair would need some sort of strain gauge to measure how much it vibrated. In contrast eyes work off physical isomerization of photoreceptor proteins, where incoming (energetic) light causes an actual structural change in the protein.

For the eyes, if you had millions of different colour receptors, you would be at a significant disadvantage in terms of two things - resolution and detection. Think of it like a digital camera's light detection sensor - each receptor is a pixel equivalent. With millions of different ones, each activated by a single wavelength, you are going to be limited in the number of each you can fit in the eye, so you limit resolution. In addition, you lose sensitivity because each receptor molecule within the cone needs a certain amount of light to be triggered, and with few total receptors you aren't going to get much signal for a nerve to work from, so your sensitivity drops. You'd also need even more of the brain to run such a system.

These are all energetic costs evolutionarily speaking. Not impossible to conceive of working in evolution terms, but just not how it evolved.