I don't agree with this at all and am and that anyone would say this.
However, I wonder if the person was approaching it with experience of families like two that I know. Both have more than 4 children, all of whom have genetic disabilities that were certainly obvious from before their third children were born. Both mothers struggled to cope with the two disabled children that they had and needed a great deal of support to cope, not to mention the usual hassles of Statementing, hospital appointments etc. Both were constantly fighting for more respite etc.
I must admit to wondering, when one family's sixth child was born, how they would manage and what would make them continue to have children, knowing that it was very likely that the next child would also have the same type of disability (not a mild one) and knowing that their family was already struggling. The authorities have continued to provide BUT I would personally be worried, financially, in this current financial climate, about how long the respite etc. will continue.
I wonder if the people saying this crap to you, OP, know families like this and have just taken it to an extreme conclusion. We all have our different limits; I think that nobody should limit the rights of someone to have a child, but sometimes I do wonder if the parents of the two families that I know might have had easier lives if they'd stuck to fewer children. The children would certainly have more time with their parents, more space and more financial security. Maybe that's what they meant.
We all know, though, that there's more to life than money and a big house .