As with everything under the sun, it depends. The two biggies that come to mind, in order of importance, are 1) which degree? and 2) which college?
I've been thinking lately how there is just a TON of talk, most of which I agree with because of what I see with my own eyes, about how most students simply waste time and money getting their degrees. For liberal arts, it's a waste in as much as the degree doesn't directly translate into a boost of income to help you pay of the criminally high tuition and expenses of attending, but also the quality of the curriculum in most (but not all) schools has been degraded to be ideological indoctrination rather than education.
For STEM fields, there are a few bad departments here and there, but so far most degrees in these directly apply as a requisite for those kinds of jobs, i.e. engineering, architecture, etc. Thus getting the degree can actually help you get a better job so you can pay off the debt, or catch back up to the money you spent if you don't take out loans for it. It's not a guarantee, but it IS a guarantee that a liberal arts degree is probably a waste of money and time these days.
The talk I referred to though is how extreme left-wing the faculty, administration, bureaucracies and even many students are at most colleges these days. This may be related to the trouble you spoke of. Not wanting to get into political talk as both left-ish and right-ish sides are needed to keep a balance in society, but want to make it clear that the "left" I speak of is very extreme currently and I would recommend a veerrry close and thorough look at any school & faculty these days before committing such an investment so it's not a waste, especially if you're an Aspie.
I say "especially if you're an Aspie" because it's extremely commonplace in these institutions to find yourself being roasted alive and berated, ridiculed and sometimes expelled just for saying something that is considered, by the powers that be, to be "offensive". For an Aspie like me who is constantly saying things the wrong way and come out twisted from what my intended utterance was supposed to be, this would be an absolute nightmare. I can't emphasize the significance of these last two paragraphs enough. I've downplayed it quite a bit. It's a really big deal.
So it's just my opinion, but it's based on quite a bit of observation, but unless you can use the degree as a prerequisite for a particular job, then I'd say you'd be much better off skipping it. It would be much more frugal to learn a skilled trade for a lot less money. Skilled labor is much more in demand than someone with a degree in, say, English Literature. In fact, my lowly technical job nets me more income than half the degreed engineers at my company. You can always learn philosophy and literature on your own - all that stuff is literally at your fingertips today.
Depending on the struggles you have that are a part of your particular Aspie package, think out what type of job will fit your personality as that matters more than it does for NT's. Just for example, my job is technical and mostly solitary so it's a good fit for me. In my case, nursing would be pretty impossible because there would be simply too many people to deal with every day. I just couldn't do that at all, but we're all different.
FWIW