I would say that 100mgs of
clomid and 20 mgs of
Nolvadex would inhibit your libido.
Even though SERMS are meant to block the estrogen receptor, they are weak estrogens, so if you take enough of them they may start to act like the more potent estrogens, thus causing the negative effects on the male libido.
Swale did say that 50mgs of
clomid was a "whalloping dose" So the combination of both these medications at the dosages you are taking may indeed be having this effect.