Marmosets show sex-linked polymorphism of color vision, whereby all males and some females show dichromatic ("red-green color-blind") vision based on two classes of photoreceptor sensitive to short or medium wavelength bands. Most female marmosets by contrast express three photoreceptor classes, one sensitive to short wavelengths and two classes in the medium-long wavelength sensitivity band. We used this 'natural knock-out' to study the organization of color vision pathways in primates. We review here results from our and other laboratories showing how the primordial dichromatic blue-yellow pathway is characterized by selective connections to short wavelength sensitive cones in the retina and that signals for blue-yellow color vision travel through an ancient part of the subcortical visual pathway called the koniocellular system. Signals serving red-green color vision by contrast are tightly linked to retinal circuits serving high-resolution spatial vision at the fovea and show little sign of specific patterns of connections with medium- and long-wavelength sensitive cones. Routine trichromatic color vision thus is based on converging signals from two quite distinct retinal and subcortical pathways.