I am not an expert in the thyroid, though have been hypo for about 25 years and have experimented with various things, (have been T3 only for 9 yrs), but unlike you am stuck on thyroid meds for life of course.
When I was first diagnosed, I was 'borderline' with a lower TSH that you, but the GP tried me on T4/thyroxine and I did improve but as my TSH still carried on rising he increased the med. This is in the UK, and these days I doubt that the GP would have allowed me to go on thyroxine until the TSH reached 10 (as you probably know the bar is lower in the US at 5 I think - but I think you are in the UK?).
In my case I did improve for just a few years, and then when I hit menopause, I got much worse and eventually ended up with a diagnosis of ME/CFS on top. My thyroid symptoms are different from the ME ones, but there is some crossover.
Anyway, what I was going to say was that assuming you haven't been taking thyroid meds that long, I think you are safe still to experiment and come off them to see how things are. In my case I cannot do that, and my thyroid gland is probably shrivelled!
I do know of people who have higher readings who then come back to normal again ). I have had some very up and down TSHs over the years since on meds, and the meds have been increased and then I had to come off the increase as it affected me adversely, and then it was normal the next time. ie I know it read as high as 15 or more on one occasion, though of course I am also on thyroid meds. Are you gluten free? I believe that some say that gluten can cause an attack on the thyroid by antibodies in a case of mistaken identity so presumably that would increase the TSH.
Certainly it is said that taking iodine can increase the TSH, so that could well be the reason (though I too thought Selenium protected that). I have always avoided iodine, but then since I am already diagnosed hypo for a long time, I don't think it would help me now.