It's not really the absolute distance that matters. There was a wormhole that connected Barrayar to Earth space, and that's what people used to get there.

Then our history got a little complicated, because the wormhole collapsed. Eventually -- try seven hundred years later of eventually -- another one was found in Barrayaran space, and we had to relearn space travel and all that sort of thing.