How do I deal with a Controlling boyfriend?

I been going out with my man for 4 years, His a Father and his kid is 12. I thought just being with him this long, I would like to start a family with him But he doesnt. his the Not that out of his gang of friends, his not going to get married. But at the same time, he tells me that he Loves me soo much and that he wants to spend the rest of his life with me without getting married and having kids.Should I let it go becuz his wasting my time or wait?