migrate crates-io-index handling to async using threads for the actual work

[syphar]

After stumbling onto some strange request::blocking errors while working on #2972 I felt like I want to be safer here. I think this is somehow related to #2937, or some update? In any case it is broken on master, so this fix blocks me deploying.

So I feel much more comfortable when we would just provide an Index struct, with only async methods, that internally uses separate threads instead of tokio's main worker or blocking threads. Neither is built for CPU load.

We're only doing the index-check every minute or so, and only once at a time, so it would probably ok-ish to use the blocking worker pool. But: then I could define the thread-name.

Also I want to see if gix has the same issues that libgit2 had, I assume it doesn't (See #847), so I migrated to keeping the repo object.

On top of that, I also moved the crates() method, which doesn't use crates-index-diff at all. It's using crates-index to get the full list of crates, and is only used in the consistency check. So I moved it there.

note

From the last all-hands there is some pending discussion / some pending changes around how we receive these events, for now we keep the git index.

[syphar]

Thinking again about this, I assume gix stuff is rather CPU bound, so using the tokio threadpools is not so good.

[GuillaumeGomez]

Thanks!