Yes leave him alone. Time is the secret ingridient to everything and since it was admittingly you that did the pushing away and it was you that had other men he has every right to be set free and find out a new direction. If time proves out that he feels he still has feelings for you and is willing to forgive then it may still be possible. But since i do not know how he feels and how deep he is hurt i can't say for sure, So give time some time to work and give him space to process everything and meanwhile IF you still want him lay off those "other men" or it is definatley over.