As the months slipped by at Holt or Chirk or both, Somerset had ample opportunity to consider his position. First, he had effectively been rusticated, which to a courtier's mind meant he was out of favour. Next, he was located in an area where the local populace were notably restive. (Not being familiar with this part of the world he may not have realised that this was pretty much par for the course for Chester and North Wales, and had been for centuries.) We can probably take it as read that he came across some Welshmen of name who were still hot for Lancaster. He may have thought on the fact that Harlech Castle was still holding out against the Yorkists, while in the North, of course, the Lancastrians were back in charge of several castles and Henry VI himself was back on English soil.
To do nothing was a risk - King Edward might have no further use for him, and in that event his future in Yorkist England would be bleak. He had many obvious enemies, including at least one or two who were exceptionally powerful. To turn his coat again also had its risks, obviously. Unless he was unusually obtuse, Somerset must have realised that - in the present circumstances - a Lancastrian restoration was a tall order. He may still have hoped that Queen Margaret would win the support of Louis XI. (In point of fact Louis had already refused to give any aid whatsoever, and Margaret was reduced to living (with her few followers) on the small amount of money her impoverished father could spare her.)
Anyway, round about Christmas 1463 Somerset departed from Wales in some haste, accompanied by a small party of followers (who may well have included his brothers Edmund - recently released from the Tower - and John of whom practically nothing is known.) After some hair-raising adventures, including one incident where Somerset was almost arrested, they eventually reached Bamburgh and made their submission to Henry VI. King Henry was doubtless very pleased to see them, as he badly needed every man he could muster. A relatively competent general like Somerset must have seemed like a Godsend.
Somerset did not sit around, but began to take such active measures as a man with very few soldiers at his disposal could. In April 1464 he attempted an ambush on John Neville, Lord Montagu, who was on his way to Newcastle to escort some Scottish envoys to York, where they were scheduled to hold peace negotiations with the Neville brothers. There was a fight, but Montagu cut his way through to Newcastle, where he promptly recruited a small force and set out to destroy Somerset. The 'armies' met at Hedgeley Moor near Morpeth on 25 April. Montagu attacked at once and Sir Ralph Percy was killed; however Somerset and the majority of men fled to higher ground and Montagu decided it was too risky to pursue with his small number of men. So he returned to Newcastle, picked up the Scots, and made his way back to York to consult with his brothers about the next step.