I bloody well hate the way some men use the old "you're not a parent" excuse to prevent their partners from expressing perfectly reasonable opinions or suggestions. You don't need to be a parent to recognise unpleasant, unacceptable, hurtful, dangerous (or whatever) behaviour though admittedly you may be a little less tolerant of it than if it were your own child. That's no reason to completely dismiss you though - and if he expects you to act as a substitute parent while he's not there - which is a big ask IMO and he should be grateful you're prepared to do this - then you have every right to have a say in what goes on.
I agree that at 3.5 kids are still babies - and their behaviour can be awful, very trying, seemingly pointless, you have to explain the same thing over and over, prone to tantrums, favourite word being "no" and so on so in that respect he's not necessarily unusual or marked out as a future delinquent! However - bad behaviour does have to be addressed, albeit in an age appropriate way, and even if you have to discipline the same sort of behaviour endlessly, the idea is that eventually the child learns - and this may take some years - that such and such is wrong/unacceptable and each time they misbehave like that so and so will happen. At any age, the key to discipline is consistency - praising good behaviour, telling off for bad, warning that continued bad behaviour will mean time out, removal of favourite toy, no ice cream or whatever's been decided is a suitable punishment. Quite obviously your DP making threats but never seeing them through is going to lead to this little boy doing what he wants - at his age he can't be expected to thoughtfully analyse his behaviour and discipline himself - he needs the adults in his life to set boundaries, so he'll learn what it takes to be a nice kid. This is even more important if his mother lets him do what he wants. Sounds like his dad is pretty much the same.
Parents like that rile me. Showing a bit of "tough love" every so often is the right thing to do, even if it sometimes doesn't feel like it. Do they really want him to be the "naughty" kid when he starts school ? The one that no other child wants to play with because they're always so insistent on getting their own way ?
I don't have to tell you that the only way to deal with this is to agree house rules, boundaries and punishments and then to apply all that consistently no matter how much SS protests. That you both back each other up in front of him and keep disagreements for when he's not there so neither of you is ever undermined. You should both have an equal say and be trusted equally - given you look after him so often - to apply discipline when needed. Anything else isn't going to work. Though he's terribly young, he's learnt that daddy's threats don't actually amount to anything - that he might be moaned at a bit, but no consequences, so to him, even if it's not a completely conscious decision, he figures that a few cross words is a small price to pay for doing as he pleases. No wonder you struggle with him as he knows his dad won't impose any repercussions. I'm not saying this little boy has sat down and analysed all this, not at all, but I'm certain that's the message that he's absorbed.
If your DP's not prepared to work with you then I don't see how this can ever work and he has a damn nerve to expect you to provide childcare without any input. I'd also be worried how in the near(ish) future things are going to pan out when your baby reaches the same age ..... will he be a very "laid back" (some would say that's a euphemism for being lazy and irresponsible) dad towards her with you being "bad cop" all the time, or will he be harder on her as she's the resident child and therefore there's no need to keep her on side to ensure continued contact ? Either way, it wouldn't be good. If he's not prepared to work together and give you equal respect and responsibility and parent consistently himself then I'd refuse to have SS over unless he was there. Why should you put yourself through that, especially with a new baby, when you're not allowed to deal with any bad behaviour in an appropriate way ?