Thank you for reaching out.
I used to panic and cut it off when people were interested in me as well. Over time, I learnt that it largely stemmed from my uncertainty at how to deal with the situation. At that point, I hardly dated and I was socially anxious and introverted. If you're up for it, I'd suggest try dating more. Forcing yourself (especially initially) to be in a romantic situation could make you more at ease and help you deal with these situations better.
It's okay not to be sure of your sexuality now. Sometimes you just have to let life happen and let your own realizations come in their own time. Don't worry about labelling your sexuality now. In fact, focus on other aspects of your life, like your job or studies etc instead of focusing on whether you're asexual or not (which is simply going to cause you a lot of anxiety and uncertainty). Over time, you will encounter more people and events that will give you a greater awareness of who you are. There is no rush.
Also, "just have sex to see what happens" is terrible, terrible advice. Have sex only when you are ready to. It will not magically reveal something to you about yourself, and if you aren't ready, I doubt the experience will be pleasant for you either. If you don't have any desire for it now, there is nothing wrong with it. Remember that it's your body, your life and your choices.
You're not dumb at all, BlueArt. Everyone goes on their own life journeys and each journey is different. Let yours unfold in its time.