I'm sad that you and Mr. Willikers aren't snuggling and cuddling yet.
I've adopted some messed up cats in the past, and, while others have said he is traumatized, I really wouldn't use that word. (Truly traumatized cats in my definition will strike at you or pee themselves with fear.) Neglected and unsocialized, yes. More than that, maybe yes, maybe no.
You really can't go by the growling. Growling is just a way for a cat to tell you to back off -- it may be fear, it may be dominance, it may be habit. Growling does NOT mean he will attack you. Most cat growls are a bluff to have you back off. Heck, Isabel growls through being groomed and she actually LIKES to be groomed. Spencer will growl when I feed him cheese and I'm sure he doesn't want me to stop!
Now, the walking with tail down and back arched is a sign of fear. It really doesn't matter because the basic problem is one of trust and confidence. Before I go on, I remember you flew out to get him (I think?). If so, what was he like with the breeder you got him from? Was she handling him? Or was he already boxed up for you? Was he a show cat?
You mentioned he slinks around at night. I'm sorry, but I haven't followed all this, but he probably should be confined to one small room and not given the run of the house. If he did live in a cage most his life, space is terrifying. Robbie lived for 3-4 years in a bathroom before I got him and it took 6 months for him to feel confident to come out of our bathroom. Once Mr. W is accepting of your presence and somewhat of your touch, then he could be let out to explore the house. This may sound harsh or silly, but it's not.
Joan made a suggestion that does work. You need to pet him for even 10 seconds (longer is better) even if he is growling. There is a tone between 'please don't' growl and the 'one more and your hand is history' growl. The tenseness of his body and his pupils are other clues. You want to pet him through the 'please don't' growl right up to the 'one more and your hand is history' growl, so that he learns two things. First, being petting isn't hurting him. Second, he's not the boss (I do wonder if his hormonal changes are messing with his mind a bit).
Joan also said something else -- she asked his you can pick him up and cuddle him. Don't pick him up unless you have to! If he's already scared of contact, having his feet off the ground is terrifying. One cat years ago I got from someone and this cat was semi-feral -- it had been living in the garage for 1-2 years. If had also been hit/kicked/poked by the owner's children, which is one reason she gave him to me. We were doing well on the 'forced' petting until someone very stupid decided to pick him up -- he went bonkers! I have scars to show you.

It also set back his trusting me a good month or more.
He can be turned around -- but it will take time and lots of love on your part.