Praise be to Allaah.
Firstly:
Your saying to your wife, It's over between us, go to your
family, is a kind of implicit divorce which does not count as such unless it
is intended as such. If you do not know what is your intention was or you
have forgotten, the basic principle is that no divorce has taken place.
Secondly:
With regard to your divorcing your wife on the second and
third occasions, the ruling is as you were advised by those muftis.
Thirdly:
You should beware of using the word of divorce lightly,
because of the consequences that result from that, which may adversely
affect the husband, his wife and his children. It is clear from the question
that you take this serious matter lightly, because you divorced her one day
and took her back, then the next day you divorced her again. This is a
transgression of the sacred limits of Allaah and a person like this deserves
to have his talaaq counted everytime, as ‘Umar (may Allaah be pleased with
him) did when the people began to take the threefold divorce lightly.
And Allaah knows best.