Or could be you have to load new textures into memory. Even though you have a nice amount of VMEM the game will not preload different textures from different areas.
It's a streaming engine, which means if you teleport to somewhere else (distance is not specified in your posts) then the game will have to take that second you described to load something that isn't a temporary texture. Higher resolution textures mean more size and more Video memory used + more time needed to load the texture.
Though judging from your original post you experience this every time, everywhere when you simply turn the camera. I doubt the game engine would need to unload textures, then again I have no idea how much your straining the engine with the additional graphic mods.