Rather late to the party, and commenting without reading the other comments in detail – so please, I may be repeating a point which someone else may have raised.
One: I have come to the conclusion that the caste system is endemic to India. It’s not an aberration; it’s what defines society. And it’s spread across all religions – a Dalit is a Dalit, whether Hindu, Christian or Muslim.
Two: Manusmriti is a law book. It’s connection to myth is very tenuous, just the mention of the Purusha Sukta of the Rig Veda at the beginning. It’s a toxic, casteist and misogynist document, which often contradicts itself. It’s also horrendously boring. (I read the whole thing in the original Sanskrit. My Sanskrit is pretty rusty, so it took me six months.)
However, I seriously doubt whether castes delineated in the document were ever seriously practised. The permutations and combinations are too numerous. What it does is, lay down the laws for the four castes – it talks about outcasts only incidentally.
Three: Even more than the Manusmriti, it is the moral justification given to caste in the Bhagavad Gita which is more revealing. Unlike the Manusmriti, this text is considered as revealed scripture by most Hindus, and it reinforces the caste model of Manusmriti (especially Chapter 12). However, the Gita is very uneven in its structure – lofty philosophy and evocative poetry mixed with didactic preaching – that one feels justified in thinking it has been bowdlerised at some point of time.