If Tarvet was top 10 individually last year, he presumably doesn't have much backup?
San Diego had a very successful season last year with a 23-4 record and almost half of those wins were won 4-0 or better. As a mid-major they will struggle to be ranked much higher than they were last season simply because in their conference they don't have the opportunity to play the higher ranked teams. The wins they get over the other teams in their conference do not give as much weight to their ranking than they would get from winning against the top teams.