You should monitor your CPU and GPU usage during these periods of low performance.
If either is pegged at 100%, then that's the bottleneck. For the CPU usage, however, you need to know how many threads the game does real work on. If it's only two, for example, then overall CPU usage of 50% would indicate a CPU bottleneck. A single-threaded game at 25% would be the same.