Done some research, asked around, and discovered beyond a doubt, 100% sure that deeds completed and quests completed on a character relate directly to load times/lag in raids.
Low level characters load fast and never lag. High level characters that have been power leveled, skipped many zones questing, and have low virtues load in fast and don't lag in game.
High level characters that have done every zone, every quest and every deed within reason, to max virtues and complete areas load slowly and lag in raids, regardless of the quality of the computer they run.
I'd like to know if turbine plans on changing the way deed/quest info is stored, so that in instances when it isn't necessary, it could possibly be stored off-site of the character... I don't really know much about game coding, but I feel like there has to be a way to do this.