Node falling behind with extensive querying

Node
1

As the title says.

I’ve experienced this on MacOS versions 5.0.6 and 5.0.7 and also on the Linux version 5.0.6.

All three instances run on machines with plenty cpu power and ram (16GB and 32GB).

I’m running a script (don’t ask why…) that executes the following commands on Concordium-client:

  1. GetBlocksAtHeight
  2. getBlockInfo
  3. getBlockSummary

As I need to perform this operation for all 4.8M blocks, I’m using a ProcessPool in Python, which executes 8 processes at the same time.

Observation 1: up until block 1M or so, I could process 1000 blocks in 30 sec. This time gradually increased to 1000 blocks in 400 sec around block 3M. I haven’t found a reason why.
Observation 2: all three nodes will run behind after 3-4 hours, usually around 10 min behind.

Any pointers?

2

Do you mean you execute concordium-client as a command? For each query?

When you say it is behind, does it recover after you stop querying?

3

Is there another (more efficient) way to query?

I need to test whether it recovers. My solution has been to restart the node, that definitely fixes it.

4

Yes, but you need to use an SDK for that.

concordium-client establishes a new connection for each query. Establishing the connection has significant overhead.

5

This observation is interesting. I will try to reproduce in a bit more controlled setting tomorrow. Thanks for these details.

6

I don’t have logs for < block 1.9M, but:
Batch 1,985 of 1,000 blocks start:1,985,000…1,986,000 finished in 140 sec.
Batch 1,986 of 1,000 blocks start:1,986,000…1,987,000 finished in 142 sec.
Batch 1,987 of 1,000 blocks start:1,987,000…1,988,000 finished in 139 sec.

and:

Batch 2,887 of 1,000 blocks start:2,887,000…2,888,000 finished in 410 sec.
Batch 2,888 of 1,000 blocks start:2,888,000…2,889,000 finished in 404 sec.
Batch 2,889 of 1,000 blocks start:2,889,000…2,890,000 finished in 420 sec.
Batch 2,890 of 1,000 blocks start:2,890,000…2,891,000 finished in 412 sec.

7

Unless I’m mistaken, there’s no python SDK? In the past you’ve helped me get started using Rust to query the node. I got that working, but never took this further, unfortunately.

I’ve built all my processes to work around output that Concordium-client commands give. I would be open to a manageable way to call an existing SDK from python, where I can call the same/similar commands to get the same/similar output?

8

I’ve started my query process this morning again (after finding that my node was still at the last block), and 4 hours later it’s behind again 5m.

Node: Concordium Dashboard

9

Further to this, I’ve taken the plunge into Rust, learning about traits, borrows, macros, etc, which seems all to be pretty different from Python…

I’ve used this example as inspiration. From what I can tell, it’s massively quicker. However, I’m struggling with this:

{
            "cost": "0",
            "energyCost": 0,
            "hash": "e8ea94651605d7140458f540f7ae61344d07b30598723ad4bf77871dc194bf49",
            "index": 5,
            "result": {
                "events": [
                    {
                        "effectiveTime": 0,
                        "payload": {
                            "update": {
                                "denominator": 176446389063,
                                "numerator": 1.40737488355328e19
                            },
                            "updateType": "microGTUPerEuro"
                        },
                        "tag": "UpdateEnqueued"
                    }
                ],
                "outcome": "success"
            },
            "sender": null,
            "type": {
                "contents": "updateMicroGTUPerEuro",
                "type": "updateTransaction"
            }
        }

This is an update transaction where the numerator is larger than the Int Type (for MongoDB), hence I’m unable to store this transaction. Now, in Python I’m able to convert the numerator to a string and store this in MongoDB.

2 questions:

  1. Is this ever going to change from Concordium’s side?
  2. Is there a way to update the numerator in Rust?

I currently have:

while let Some(mut tx) = transactions.next().await.transpose()? {`
match details {
BlockItemSummaryDetails::AccountTransaction(_) => {}
BlockItemSummaryDetails::AccountCreation(_) => {}
BlockItemSummaryDetails::Update(x) => {
## Do something here to update the numerator in the payload...
}