The difference is just what I said: on the one hand, you're dealing with volume, and on the other, you're dealing with pitch.
There's nothing wrong with your ears—your brain is just wired to handle the particular prosodic devices of English, so it perceives Chinese words through that filter. With prolonged exposure, you would start picking up the difference.
For example, I decided to make Feayran tonal some time before I started learning Chinese. In lessons I wrote up for some friends, I explained the different tones in terms of English stressed and unstressed syllables, because I thought they were the same. But now, after speaking Chinese for several years, the difference between a Chinese high tone and an English stressed syllable is very obvious to me—they sound very different. Once I started hearing the difference, I could start pronouncing Feayran with actual pitch-based tones.
Which is why linguists distinguish pitch accent from dynamic accent.
Ah, I see. Perhaps that's why also my chinese friend said I pronounced it right, because she percieved my stress changes as tonal changes, like I did the reverse? That makes sense.
I wouldn't believe you, if it weren't for the fact that just last year, I wouldn't have been able to tell the difference between the sounds of dental and alveolar "t, d, etc..."s, "t" and "Aspirated t", and the list goes on and on. Also, don't forget half the vowels of IPA sounded the same to my american ears until I "learned" to spot the difference.