GetAccountInfo directly from node db

Clients
1

I’m currently retrieving AccountInfo by calling concordium-client raw GetAccountInfo with an account_id, but was wondering if there is a more direct (and possibly quicker) way of getting account info, by queuing the node database directly. If so, where should I start?

2

Hi,

No you are correct with your approach. This is the easiest and fastest way at the moment.

Best
Emil

3

Note that you can speed up queries by using one of the SDKs and

  • making parallel queries
  • reusing the same connection for different queries.

If you make queries with concordium-client it will make a new connection to the node each time you invoke it which adds significant overhead for a fast query like GetAccountInfo.

4

You’re going to make me learn Rust, aren’t you? :grinning:

5

Alright, biting the Rust bullet (assuming there’s no python SDK), but where to start? I found this example which I’m hoping to adapt for my needs.

I’m sure with enough effort I’ll be able to grok this, but right now I’m struggling with the dev environment to get this working. Is there documentation available what I would need to do/install to get above example code working?

6

Not directly. Note that there is a NodeJS SDK available as well, if you would find that more familiar GitHub - Concordium/concordium-node-sdk-js: Wrappers for interacting with the Concordium node.

Regarding that example, it is part of the SDK so the requirements are

  • install rustup https://rustup.rs/
  • using rustup install the rust toolchain. I recommend 1.53 since that’s the version we test with. You can do this by rustup default 1.53
  • then from the root of the repository build the example. You do this by using cargo build --release --example list-account-balances

This should build the binary. The binary is located in ./target/release/examples/list-account-balances and you should be able to run it by doing

./target/release/examples/list-account-balances --node http://localhost:10000 --out file.csv

or obvious modifications of arguments.

7

Thank you so much for this. Greatly appreciated.

So far I’ve installed rustup (using defaults), installed the toolchain (using the command provided), cloned the repo and from the root tried to build using cargo build --release --example list-account-balances.

This leads to the following error:

❯ cargo build --release --example list-account-balances
error: failed to get `aggregate_sig` as a dependency of package `concordium-rust-sdk v0.1.0 (/Users/sander/Developer/CCD/concordium-rust-sdk)`

Caused by:
  failed to load source for dependency `aggregate_sig`

Caused by:
  Unable to update /Users/sander/Developer/CCD/concordium-rust-sdk/concordium-base/rust-src/aggregate_sig

Caused by:
  failed to read `/Users/sander/Developer/CCD/concordium-rust-sdk/concordium-base/rust-src/aggregate_sig/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

What could have caused this? I’m not yet well versed in interpreting Rust error messages.

8

Scratch that, I forgot to include the git submodules…

1 Like
9

Yeah, for other readers, the submodules should be checked out, so git submodule update --init --recursive.

1 Like
10

Hmm, the example in Rust actually ran 60% slower (16 min) than my python code (10 min), doing roughly the same? Not what I was expecting at all.

Start: 2022-03-12T20:06:33.866663+01:00
Listing accounts in block 6aa247b76a8436767cae30183666140bd69ebbccfdd399709c947d22eaf812e5.
There are in total 82709 accounts in block 6aa247b76a8436767cae30183666140bd69ebbccfdd399709c947d22eaf812e5.
228 of these accounts are bakers, and 75504 are initial accounts.
Total amount of CCD is 10747895321.720982.
Total amount of staked CCD is 8512255780.116415, which amounts to 79.20%.
End: 2022-03-12T20:22:19.607777+01:00
11

I am not exactly sure what your python script does. But basically all of the time is spent waiting for the node when running the script which should be visible by the fact that CPU use should be very low.

Note that this example does not do parallel queries. Depending on how your node is set up, you can probably get 8 times or more speedup by making use of that.

12

My python script takes the the account list at a block and then iterates over this list, requesting accountInfo for each account. I’m using threadPoolExecutor, which utilizes my 8 cores (which indeed are all fully used for 10 min).

My assumption was that using Tokio (with flavor=multi thread) in the example I was using parallel requests, but again, I know basically nothing of rust at this moment.

What would I need to change in my node (setup) to facilitate parallel queries?

13

Tokio indeed allows for parallel requests, but you have to tell it to do that. That example is written as sequential. And since that specific example is really not CPU bound, but IO bound, the node being the bottleneck, using the multi-threaded executor does not actually matter much.

I updated the example to make parallel queries, see Update list-account-balances example to make parallel queries. · Concordium/concordium-rust-sdk@01b00f7 · GitHub (full source in the usual place concordium-rust-sdk/list-account-balances.rs at main · Concordium/concordium-rust-sdk · GitHub )

The main change is that I first schedule a bunch of futures to be executed, and then execute them in parallel ( Update list-account-balances example to make parallel queries. · Concordium/concordium-rust-sdk@01b00f7 · GitHub )

The default is 8 queries in parallel, use --num to select some other value. CPU use should still be very low, since, again, the node is the bottleneck.

14

Thank you, this will serve as a perfect starting point to migrate my Python work over to Rust. With num=16, I got it down to 4 min with low CPU usage.

Now all I need is a way (on MacOS) to automate the start and stop of the node, without admin intervention, and then I can auto update Concordium Explorer Statistics. I’ve tried removing the “with admin privileges” in the AppleScript, but that did not work.