I'm still trying to understand myself and I'm starting to think that having a special interest is more of a need than just a want or a like.
For a long time, I thought my anxiety was tied to my own perception of my performance. If I felt like I Accomplished something on any particular day, then I could go to bed satisfied and happy. When I didn't feel like I accomplished anything significant, I would feel anxious and worthless at the end of the day.
Then I started supplementing my job work or my house work with projects related to my special interest (math). I would always have some math problem or mathematical curiosity to think about. When my work didn't provide that sense of accomplishment, I would look for it in my special interest.
Eventually, I came to realize that:
1) I got good reviews at work, even when I didn't feel like I accomplished much. This means that what I was calling "accomplishment" isn't what others would call "accomplishment."
2) My enjoyment from math came from working on the problems, not necessarily from solving them. I have worked on so many open math problems (unsolved problems) and haven't solved a single one - but I've enjoyed learning about them and seeing how far I could get. In fact, if I chose a problem that was too easy, and I solved it, I would feel disappointment because that meant that I would have to look for another problem to think about.
Putting those two things together, my anxiety (when not not doing my special interest) and satisfaction (when doing my special interest) have nothing to do with accomplishment or achievement. It's the activity itself that drives the anxiety away.
I'm beginning to think that, yes, math is very interesting to me, but there's more to it than that. I need time with my special interest as much as I need quiet time to recharge after social events, or as much as I need food and sleep. There is some need for stability, certainty, and a comfort-zone that my special interest provides.
It's not just an interest, it's a need.