What's up with this guy? Am I just a friend to him?

He probably doesn't want it to seem like he is chasing you. Or thinks he would be interrupting you talking to someone else. Has he asked you out yet? If not, he may be really afraid of rejection and unsure how you would answer. If you want him to be more direct, you may have to boost his confidence some.