Have you ever...
That's a really fascinating question. A lot has been written on accents actually.
The basic idea is that babies can hear distinguish all possible sounds humans can make. But our first language will restrict this and apply a filter to group similar sounds into a same unit (phoneme) so that we can understand people even if they don't produce every sound the same.
This happens really quickly in a human's life, like by the age of 6 months it's set. After that, it is really hard to learn to distinguish sounds that have been categorized as another sound.
When we speak, we use our filter, for example we will use the closer sound in our language to produce a sound that doesn't exist. That's explains why some people learning English when try to do the "th" sound in "this" will say [z] if they're French native speakers, or [d] if they have a Germanic language... or many more.
If you'd like to read more about this filter was researched by Soviet linguist Nikolai Trubetzkoy and the experiments with babies and their phonological filters were researched by Flege, you can search for Speech Learning Model.
This explains why we have an accent, because after a certain age (not determined perfectly and it depends on people), we simply won't be able to hear, and if we learn to hear and distinguish, we may not be able to produce sounds exactly like native speakers.
But we should also keep in mind that everyone has an accent, even in our own native language :)