Thinking of how we feel about the IRS, we can understand why Matthew was not well liked. Then add on the facts we see below, which is that tax collectors could line their own pockets with whatever they could get away with ... well, I feel the crowd's astonishment when Jesus calls Matthew to follow him.
Matthew was a well-hated man. Tax-gatherers can never be a popular section of the community, but in the ancient world they were hated. People never knew just how much they had to pay; the tax-collectors extracted from them as much as they could possibly get and lined their own pockets with the surplus that remained after the demands of the law had been met. Even a Greek writer like Lucian ranks tax-gatherers with "adulterers, panderers, flatterers and sycophants." Jesus wanted the man no one else wanted. He offered his friendship to the man whom all others would have scorned to call friend...I'd like to note one other thing here. Mary Healy in The Gospel of Mark: Catholic Commentary on Sacred Scripture points out that although the Pharisees disapproved of Jesus, they questioned his disciples. Jesus answers because he overhears. Is this because they lack the courage to confront Jesus or because they are trying to shake the disciples' faith? An interesting point and one to consider when we ourselves are questioned similarly, as is all too common these days.
Of all the disciples Matthew gave up most. He literally left all to follow Jesus. Peter and Andrew, James and John could go back to the boats. There were always fish to catch and always the old trade to which to return; but Matthew burned his bridges completely...
The odd thing is that Matthew's reckless decision brought him the one thing he can least have been looking for -- it brought him immortal and world-wide fame. All men know the name of Matthew as one forever connected with the transmission of the story of Jesus.
The Gospel of Mark (The Daily Bible Series*, rev. ed.) by William Barclay
* Not a Catholic source and one which can have a wonky theology at times, but Barclay was renowned for his authority on life in ancient times and that information is sound, as are many of his general reflections.