I think what's more of an indicator is your own size, and your DH's. Are you both small? If so, your DS is likely to be small too. My 23mo DS is quite big for his age (was around the 90th% for height last time we had him measured and has size 6 feet) but I'm under no illusion that he will be tall. I'm 5'3" and DH is 5'8" so at some point I'm sure DS will be overtaken by his peers.
You're obviously worried though so why not speak to your GP/HV to see if you can have any growth issues ruled out. At least then you can put your mind at rest.