You don't mention how old he is, so, to me, that might be a factor. Maybe he really isn't as ready to be away from home for that long (and maybe either doesn't want to say it, or else doesn't even recognize it himself).
Something else, though - and that's that I'd be more concerned with addressing the fact that he says he thinks the other kids will hate him. Again, depending on his age, I don't know if I'd even encourage him to go if he wasn't completely enthusiastic about it and acting like he was completely sure he wanted to go. Maybe it depends on if he's really "completely convinced", only "kind of worried about the issue".
Just throwing out some "maybe's", but maybe he liked the idea of "elite" when he applied (and before he had to deal with the realities of having been accepted).
To my thinking (and it's only my thinking), one of "maybe's" could be that he's just not someone who really wants to go to that particular thing (maybe because he isn't someone who comfortably fits into some types of programs/activities).
I mean.. to me, if he isn't completely thrilled with having the opportunity then there's something about it that's not right for him. On the one hand, if he's only mildly nervous about whether he'll find friends then simple reassurances that he most likely will find at least a friend or two ought to do. If he's more than "normal, mildly-nervous" I think I'd make sure he was completely comfortable with the idea of saying he decided he didn't really want to go after all.
One question might be what, exactly, will he get from the program (if he's already gifted in math/science) that he really needs anyway (or that he can't pursue on his own anyway)? I suppose my "thing" is that if I had a child with that kind of issue I might try to find shorter-time/distance activities that might help him get a little past any social problems he has,
For a kid (regardless of age) to even consider that the other kids are going to hate him... that's a real concern because the world is full of gifted children who don't have social problems.
With only the information provided here, I suppose if it were my son (and if it were at all possible) I'd let him know he could go, see if he liked it OK enough, and let him know that if he hated it after a couple/few days he could come home.
Inside, he either really wants to go or doesn't. I'd make sure he has a face-saving "out", if nothing else. If he's ready and wants to go he will.