We present a multiple regression analysis of time series of the leading modes of variability in high latitudes, the Northern and Southern Annular Modes (NAM and SAM). The potential forcing factors we investigate include solar activity and stratospheric (volcanic) aerosol, the quasi-biennial oscillation, the El Niño-Southern Oscillation (ENSO) and stratospheric ozone concentrations. We investigate levels from the surface to the middle stratosphere and we also conduct a similar analysis of the North Atlantic Oscillation (NAO) in surface pressure. We find a significant trend in SAM at all levels and in surface NAM but not in the NAO. There is some correlation of ENSO with both the SAM and NAM, with warm events being associated with weakening of the polar vortices. Strong volcanic aerosol signals are found with different signs in the two hemispheres. There is a clear solar influence on the NAO but no statistically significant signals of solar forcing are found in either the NAM or SAM. The QBO is not detected except, marginally, in the NAM in winter. However, when a new index, the product of the solar and QBO indices, is used in place of the two factors individually then a good correlation is found throughout the atmosphere in the SAM, and at the lower levels in the winter NAM. The different results for the NAO and surface NAM lead us to conclude, speculatively, that a (solar) stratospheric influence on the troposphere may well be through two different routes. The first of these is the influence of low latitude lower stratospheric heating on the Hadley circulation and mid-latitude eddies and thus on the NAO; the second is modulation, by a combination of solar and QBO forcing, of the polar stratosphere which influences the annular modes in both the northern and southern hemispheres.