David
As far as I know there were not very many Caldwell's in Annandale. I have read a great number of Scottish History Books and I have rarely seen the Caldwell surname mentioned - and usually in the context of Servant. That there were Caldwell's around there is no doubt and there are obvious non-history book references to them.
I am too long out of Scotland but I would hazard that Annandale is hardly Galloway.
I would only be too happy to accept that there was a "Coldwell" estate in Galloway or Annandale but it appears so far that this legend is confined completely to the USA.
I was pleased when David of Manitoba came up with some information from another source regarding some French Caldwell's that settled in Ayrshire.
It has been posted on this site and gives a little comfort to the French Caldwell's and a possible "Coldwell" estate.
Firstly the places mentioned in Scotland are in Ayrshire not Annandale. The predominant location of Caldwell ancestors in Scotland runs from Ayrshire through Renfrew and partly into Lanark. My research has found Caldwell's mentioned in the 100's and later in Wigton, Dumfries, and the Lothians. A normal dispersion for lowland Scots but the vast majority are in the areas that I first mentioned.
I took the dates and locations of the Caldwell family in France looking up the locations in a map and studying some history texts as to what was happening politically at the various times.
The early dates relate to a period when Gascony in Southern France was part of England. The first Caldwell appeared about the time Edward I of England spent two years there on his way back from a crusade. Perhaps this Caldwell was in his military train and stayed in the garrison of Tartas. A wife Johanna had an English-sounding name. Gascony was one of the stongholds of the Hugenots. It was lost to the English crown in the reign of King John.
The areas where the later French Caldwell's (of the family mentioned) lived were in northern France bordering Belgium. The towns mentioned were either Hugenot stongholds or close to Hugenot areas.
The Hugenots were leaving France well before the Revocation of the Edict of Nantes because of persecution.
One of the industies that the Hugenots wer famous for was lacemaking. The village of Darvel about 10 miles inland from Kilmarnock on the Irvine River is famous for lacemaking. Whether this was started by Hugenots or not I do not know. I do know that Nottingham in England which even more famous as a lacemaking centre is also in the English Caldwell country. I don't think there is any connection though.
If the French Caldwell's came to Scotland I am sure that they did not come in sufficient numbers, or early enough, to create the Caldwell dynasty. It is a curious co-incidence and it is likely perhaps that one branch of Scottish Caldwell's may well have come there from the English family via many many generations in France.
They couldn't be anything other than very distant cousins at best.
I think there are some serious clues there and it would be appropriate for some interested researcher with French to pursue the matter further. It might (and should) be of extreme interest to those US Caldwell's who subscribe to the French connection myth.

Here is a lead - go for it - I am sure that there are serious tales to make out of the information there. If you can find some more supporting facts then share them around - we are all interested.

For my part I am content to continue to believe that my Caldwell ancestors have been Scots from antiquity.

TomC