I was doing a BSc(Comp Sci) at uni for a few years.. got a job as part of the course and am still with that company (kinda) now, never got around to finishing it.
I went back at one point, but was disgusted by the fact that some of the lecturers were actually asking me for help, and didn't go back. I'm not a programmer - I went more a networking/sysadmin type roll. However, one thing that I did pick up from doing the degree was an understanding of the concepts of programming, which is really probably the most valuable thing - understand how to piece together a program to do what you need at a higher level - program design, rather than coding. Once you know what you want to do, it goes to coding and picking up the syntax of whateveer language is being used. You won't remember that unless you use it regularly. I've studied more languages than I can remember the names of, and I remember only bits and pieces of the actual syntax. But it's the concepts that you retain.. looping, boolean tests, classes, all that sort of stuff.
As a result, I can debug code pretty well - I can look at code, work out what it's doing and how it's doing it, then with maybe a little reference find the syntax I need to get it to work how I want it. Coding from scratch takes a little more work.
What it really comes down to though, is as people have said: If you're genuinely interested in the stuff, then you can do well. If you're just doing it 'because', and are struggling through, then a career in programming might not be for you, but that's something you've got to decide for yourself. One of the guys that works here is a Perl guru, on damn good pay. He's like that because it's what he does. He goes home and 'plays' by writing new programs.. wrote a bittorrent proxy, for example. The guy hasn't studied anything officially. In fact, he never even finished high school.
He's just interested in what he does, and enjoys his work.