Concordium node version 7.0.5 is released for testnet.
The 7.0.5 release fixes a bug in the handling of smart contract names that could cause the node to crash.
This is a critical bug fix, and node runners should update as soon as possible.
1 Like
Hello, we upgraded to 7.0.5 and get this error during start. Is there fix to this? Sep 30 16:26:28 provenance-tags-concordium-testnet-node systemd[1]: Started Concordium Testnet Node.
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node::utils] Starting up concordium_node version 7.0.5!
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-4221332d34e1694168c2a0c0b3fd0f273809612cb13d000d5c2e00e85f50f796/data
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-4221332d34e1694168c2a0c0b3fd0f273809612cb13d000d5c2e00e85f50f796/config
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node::utils] Network: enabled
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node::utils] Log level: info
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node::p2p::maintenance] My Node ID is 9637d88882e91206
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node::p2p::maintenance] Listening on 10.164.0.11:8889
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node] Starting consensus layer
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Sep 30 16:26:28 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
^[Sep 30 16:26:29 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: concordium-testnet-node-7.0.5: DatabaseInvariantViolation “no version data was found, but expected {databaseVersion: 1, protocolVersion: P7}”
Sep 30 16:26:29 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[1878]: concordium-testnet-node-7.0.5: warning: too many hs_exit()s
Sep 30 16:26:29 provenance-tags-concordium-testnet-node systemd[1]: concordium-testnet-node.service: Main process exited, code=exited, status=1/FAILURE
Sep 30 16:26:29 provenance-tags-concordium-testnet-node systemd[1]: concordium-testnet-node.service: Failed with result ‘exit-code’.
This error suggests that there was a problem in the protocol 6 to 7 migration resulting in the database not being properly initialized. It would be helpful to see earlier logs to try to resolve the root cause of this. It would also be helpful to know a few details such as which node version you were upgrading from, whether it was before or after the protocol update was effective on testnet (around 3pm CEST yesterday) and if there were any other issues that occurred around that time (in particular, the protocol update can take a couple of minutes to run, and if the node process was killed in that time, we might see this result).
The following steps should get over the database issue.
- Stop the node if it’s running: e.g.
sudo systemctl stop concordium-testnet-node - Remove the corrupted treestate database (only the
-4database should be removed): e.g.sudo rm -r /var/lib/concordium-4221332d34e1694168c2a0c0b3fd0f273809612cb13d000d5c2e00e85f50f796/data/database-v4/treestate-4 - Restart the node: e.g.
sudo systemctl start concordium-testnet-node
Please let me know if this resolves the issue and send further logs if you have them so we can try to prevent this issue in future.
Thanks! We reverted (using wget + apt install) to the original version 6.3.0, which is running at the moment. Initially, we upgraded to 7.0.4, which started fine but crashed after some time. We then attempted to upgrade to 7.0.5, but it resulted in an error. Reverting to 6.3.0 was successful.
Downgrading from 7.x to 6.x is potentially problematic, because the version 7 node compiles smart contracts differently from the version 6 node. In any case, the 6.3.0 node cannot keep up since the protocol update, so I recommend upgrading to 7.0.5.
Do you have any logs around when 7.0.4 crashed?
No logs are available for 7.0.4. I’ll make another attempt with 7.0.5 later tonight and will remove the database first, as mentioned earlier. I’ll update you on the outcome.
We did the commands above and installed 7.0.5 and see this now: journalctl -u concordium-testnet-node.service -f
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::utils] Application data directory: /var/lib/concordium-4221332d34e1694168c2a0c0b3fd0f273809612cb13d000d5c2e00e85f50f796/data
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::utils] Application config directory: /var/lib/concordium-4221332d34e1694168c2a0c0b3fd0f273809612cb13d000d5c2e00e85f50f796/config
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::utils] Network: enabled
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::utils] Log level: info
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::p2p::maintenance] My Node ID is 9637d88882e91206
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::p2p::maintenance] Listening on 10.164.0.11:8889
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node] Starting consensus layer
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::plugins::consensus] Starting up the consensus thread
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::consensus_ffi::consensus] Starting up the consensus layer
Oct 01 14:10:44 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [WARN concordium_node::consensus_ffi::ffi] GlobalState: Block state file exists, but tree state database does not. Deleting the block state file.
Just to confirm: 1d30923ca9dc347706e0c3ed9b9e24a6d14c80c50eb arrived at 2024-10-01 14:24:29.608994791 UTC. Processed in 0.00880825s.
Oct 01 14:24:29 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::consensus_ffi::ffi] Konsensus: Block f3645406f22139b8a03b146f8e0ac23ba3f7e3224ae2bf4a7bcf35754001b957 (round 7082) finalized at height 6762
Oct 01 14:24:29 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::consensus_ffi::ffi] Konsensus: Block f59c5b493a46a42322c5953d2087a786346c4b453d08198423038d767ad918e6 received.
Oct 01 14:24:29 provenance-tags-concordium-testnet-node concordium-testnet-node-7.0.5[12508]: [INFO concordium_node::con
From those logs, it looks like it’s working normally.
Yes - just to confirm that the commands worked