Eric Gunnerson made a post related to performance of generics today which has some merit to it (but isn't that the case for about anything from him?!)
Performance is rarely dominated by small decisions, such as whether you use an ArrayList or a List<int> to store your data. It's usually dominated by algorithmic concerns - how you process data, how much redundant processing you're doing, etc. My experience is that the ability to be able to address these concerns is fairly well correlated with the resiliency of your code to change, which usually comes down to how clean and understandable the code is. If you have clean code and good tests, it is probably feasible to refactor your code to improve performance once you find out where the performance issues are.
I totally agree. I sometimes laugh when I hear developers argue about X is faster than Y, when both X and Y are such insignificant parts of their code. Sure, X might be faster than Y, but in the context used does it really make a difference? Of course, the big design decisions are where it is at. I'm not saying that small performance gains are unimportant, but the focus should be performance of the bigger picture. As Rico says, take the time to understand the performance needs of your customer and make that your focus. Otherwise you might get headed down the road of premature optimization.
Oh yeah, that and make sure you're a smart enough developer that the small performance gains come as second nature. Know the right way to code to get the best “general performance” and avoid stupid habits that will slow down your code.