That's more a masculine gender role issue, than a problem with the word.The operative word there though is 'little', try saying that about boys over the age of four. What boy of 10/11 etc wants to be called sweet? I'm not sure parents would be happy about having a 'sweet' boy of 12, many would be somewhat insulted that it was the best you could say about a lad.