Most of the gems you find are so-called "fake" gems. Many people you see in matchmaking are not running a ritual at all at the moment you see them. Instead of displaying all these totems as empty, fake gems are added by the game for you to steal. These gems are usually low in value (up to 10k), although very occasionally you may see one up to 100k.
Further, if you're level 61, you can see all these high level players in matchmaking But some of them are online a huge amount of the time every day, playing many hours, which makes it less likely you'll see them.
The best advice I can give you for finding bigger gems is to simply skip more. Increase your pace of skipping too, but making sure you keep a speed for which you won't accidentally skip past a base you'd like to raid. Many top players skip millions of gold every day, spending several hours just tapping that button and halting only when they see a base with big gems.
If you found someone in Invisius with 1M+ delta, you're a little unlucky. Usually it is possible to win Invisius with a lower amount (say, 300k or so).
- AGoneCZ likes this