Node isn't running properly

petlan · September 13, 2022, 7:08am

Hi,

My node (testnet) have stopped working. It is running in a virtual environment that isn’t online all the time, so when I start it, it will have to do some catching up, before it is up-to-date with current affairs of the chain.
It has been working fine this way though and is only used for developmet - now it stopped working though.

I get the following message when queried for status:

sudo systemctl status concordium-testnet-node.service

 concordium-testnet-node.service - Concordium Testnet Node
     Loaded: loaded (/lib/systemd/system/concordium-testnet-node.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/concordium-testnet-node.service.d
             └─override.conf
     Active: activating (auto-restart) (Result: exit-code) since Tue 2022-09-13 09:03:09 CEST; 3s ago
    Process: 5551 ExecStart=/usr/bin/concordium-testnet-node-4.3.1 --config-dir ${STATE_DIRECTORY}/config --data->
   Main PID: 5551 (code=exited, status=1/FAILURE)
        CPU: 399ms

Any pointers to what I can try to get it back up?

Zoltan · September 13, 2022, 7:47am

Hi Petlan
Can you provide us your log file?

Regards,
Concordium Tech Support

petlan · September 13, 2022, 7:56am

Hmm I can’t seem to find the location of the log file - where should I look for this file?

Zoltan · September 13, 2022, 8:06am

Hi Petlan

on windows should be here. “C:\ProgramData\Concordium\Node Runner\testnet\logs”
on the other platforms something similar, find the node runner folder and choose testnet\logs.

PabloZ · September 13, 2022, 8:22am

you have deb installer, right ? you can check latest logs with this command

sudo journalctl -u concordium-testnet-node -f

petlan · September 13, 2022, 8:30am

Yeah an I was just looking back in older posts to find this command

It produces this result:

sudo journalctl -u concordium-testnet-node -f

Sep 13 10:27:00 peter-VirtualBox concordium-testnet-node-4.3.1[11366]: [INFO  concordium_node] Starting consensus layer

Sep 13 10:27:00 peter-VirtualBox concordium-testnet-node-4.3.1[11366]: [INFO  concordium_node::plugins::consensus] Starting up the consensus thread

Sep 13 10:27:00 peter-VirtualBox concordium-testnet-node-4.3.1[11366]: [INFO  concordium_node::consensus_ffi::consensus] Starting up the consensus layer

Sep 13 10:27:01 peter-VirtualBox concordium-testnet-node-4.3.1[11366]: concordium-testnet-node-4.3.1: too few bytes

Sep 13 10:27:01 peter-VirtualBox concordium-testnet-node-4.3.1[11366]: From:        demandInput

Sep 13 10:27:01 peter-VirtualBox concordium-testnet-node-4.3.1[11366]: CallStack (from HasCallStack):

Sep 13 10:27:01 peter-VirtualBox concordium-testnet-node-4.3.1[11366]:   error, called at src/Concordium/GlobalState/Persistent/BlobStore.hs:226:27 in concordium-consensus-0.1.0.0-9jU8x94HOPtztdj4f8Bui:Concordium.GlobalState.Persistent.BlobStore

Sep 13 10:27:01 peter-VirtualBox concordium-testnet-node-4.3.1[11366]: concordium-testnet-node-4.3.1: warning: too many hs_exit()s

Sep 13 10:27:01 peter-VirtualBox systemd[1]: concordium-testnet-node.service: Main process exited, code=exited, status=1/FAILURE

Sep 13 10:27:01 peter-VirtualBox systemd[1]: concordium-testnet-node.service: Failed with result 'exit-code'.

Sep 13 10:27:21 peter-VirtualBox systemd[1]: concordium-testnet-node.service: Scheduled restart job, restart counter is at 320.

Sep 13 10:27:21 peter-VirtualBox systemd[1]: Stopped Concordium Testnet Node.

Sep 13 10:27:21 peter-VirtualBox systemd[1]: Started Concordium Testnet Node.

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node::utils] Starting up concordium_node version 4.3.1!

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node::utils] Application data directory: /var/lib/concordium-4221332d34e1694168c2a0c0b3fd0f273809612cb13d000d5c2e00e85f50f796/data

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node::utils] Application config directory: /var/lib/concordium-4221332d34e1694168c2a0c0b3fd0f273809612cb13d000d5c2e00e85f50f796/config

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node::utils] Network: enabled

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node::utils] Log level: info

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node::p2p::maintenance] My Node ID is c59f4a2b589d67e7

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node::p2p::maintenance] Listening on 192.168.1.23:8889

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node] Starting consensus layer

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node::plugins::consensus] Starting up the consensus thread

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: [INFO  concordium_node::consensus_ffi::consensus] Starting up the consensus layer

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: concordium-testnet-node-4.3.1: too few bytes

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: From:        demandInput

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: CallStack (from HasCallStack):

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]:   error, called at src/Concordium/GlobalState/Persistent/BlobStore.hs:226:27 in concordium-consensus-0.1.0.0-9jU8x94HOPtztdj4f8Bui:Concordium.GlobalState.Persistent.BlobStore

Sep 13 10:27:21 peter-VirtualBox concordium-testnet-node-4.3.1[11392]: concordium-testnet-node-4.3.1: warning: too many hs_exit()s

Sep 13 10:27:21 peter-VirtualBox systemd[1]: concordium-testnet-node.service: Main process exited, code=exited, status=1/FAILURE

Sep 13 10:27:21 peter-VirtualBox systemd[1]: concordium-testnet-node.service: Failed with result 'exit-code'.

Sep 13 10:27:41 peter-VirtualBox systemd[1]: concordium-testnet-node.service: Scheduled restart job, restart counter is at 321.

Sep 13 10:27:41 peter-VirtualBox systemd[1]: Stopped Concordium Testnet Node.

Sep 13 10:27:41 peter-VirtualBox systemd[1]: Started Concordium Testnet Node.

Sep 13 10:27:41 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node::utils] Starting up concordium_node version 4.3.1!

Sep 13 10:27:41 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node::utils] Application data directory: /var/lib/concordium-4221332d34e1694168c2a0c0b3fd0f273809612cb13d000d5c2e00e85f50f796/data

Sep 13 10:27:41 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node::utils] Application config directory: /var/lib/concordium-4221332d34e1694168c2a0c0b3fd0f273809612cb13d000d5c2e00e85f50f796/config

Sep 13 10:27:41 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node::utils] Network: enabled

Sep 13 10:27:41 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node::utils] Log level: info

Sep 13 10:27:41 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node::p2p::maintenance] My Node ID is c59f4a2b589d67e7

Sep 13 10:27:41 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node::p2p::maintenance] Listening on 192.168.1.23:8889

Sep 13 10:27:42 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node] Starting consensus layer

Sep 13 10:27:42 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node::plugins::consensus] Starting up the consensus thread

Sep 13 10:27:42 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: [INFO  concordium_node::consensus_ffi::consensus] Starting up the consensus layer

Sep 13 10:27:42 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: concordium-testnet-node-4.3.1: too few bytes

Sep 13 10:27:42 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: From:        demandInput

Sep 13 10:27:42 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: CallStack (from HasCallStack):

Sep 13 10:27:42 peter-VirtualBox concordium-testnet-node-4.3.1[11415]:   error, called at src/Concordium/GlobalState/Persistent/BlobStore.hs:226:27 in concordium-consensus-0.1.0.0-9jU8x94HOPtztdj4f8Bui:Concordium.GlobalState.Persistent.BlobStore

Sep 13 10:27:42 peter-VirtualBox concordium-testnet-node-4.3.1[11415]: concordium-testnet-node-4.3.1: warning: too many hs_exit()s

Sep 13 10:27:42 peter-VirtualBox systemd[1]: concordium-testnet-node.service: Main process exited, code=exited, status=1/FAILURE

Sep 13 10:27:42 peter-VirtualBox systemd[1]: concordium-testnet-node.service: Failed with result 'exit-code'.

PabloZ · September 13, 2022, 8:36am

you have to follow this instruction:
https://developer.concordium.software/en/mainnet/net/nodes/troubleshoot-ubuntu.html

Basically, you need to stop the node, go to the folder with the blockchain database and delete latest blockstate-.dat file and the treestate- directory. After this start the node and wait for catch-up.

petlan · September 13, 2022, 8:55am

This was what I did the last time around as well.
The latest files/folders were xxx-1, so I removed them and restarted the node.
Had to also restart the collector to get the node to show on the dashboard.

Any idea why this keeps happening?

Could it be related to the fact that the node isn’t running all the time and when it’s trying to catch up, something goes wrong?

I have checked the storage on the virtual system and it has not run out of storage space.

PabloZ · September 13, 2022, 9:09am

This usually happens after a node has been stopped incorrectly. To stop a node correctly, use the sudo systemctl stop concordium-testnet-node command. You can also use the concordium-node-stop binary (this binary was previously distributed with the docker archive).
I made a service that gives the user an archive with a fresh database every day. To do this, I have to stop the node every day, then restart it after some time. So, for all the time my service has existed, I’ve never had any problems with database corruption. I stop the node simply by using the usual command sudo systemctl stop concordium-testnet-node and the database is not damaged.