Levi got up, left everything, and followed him. Then Levi gave a great banquet for him in his house; and there was a large crowd of tax collectors and others sitting at the table with them (Lk 5,28-29)
Following someone means leaving something else. Following Jesus means leaving everything else. And Levi has done the right thing. He left everything and followed Jesus. Following Jesus was not the problem for the Pharisees or leaving everything behind was also not the problem.
The problem was in eating and drinking with the tax collectors and sinners. The Pharisees were against Jesus regarding his table fellowship. But Jesus came into this world to build up the relationship. Jesus knew well that the chosen people of God were divided among themselves and there were groups among them.
Jesus chose Levi to be one of his disciples. Surely this would not have pleased the Pharisees. Not only he chose a disciple from this group but also he ate and drank with them. His second action was difficult for the Pharisees to accept. Eating and drinking would be equal to having friendship and relationship with them. Table fellowship was sacred for the Jews.
Jesus always takes the side of the suffering people and not the morally good people.