We encounter a “ResourceExhausted” node error when pushing a large number of transactions. The error message states, “Too many concurrent requests.” According to the documentation, this issue can be resolved by increasing the --grpc2-max-concurrent-requests (CONCORDIUM_NODE_GRPC2_MAX_CONCURRENT_REQUESTS). For testing purposes, we have set this parameter significantly higher than what we anticipated, at 100,000. Are there any other parameters we should adjust to mitigate this issue?
If you are getting “Too many concurrent requests” then the parameter you are hitting is CONCORDIUM_NODE_GRPC2_MAX_CONCURRENT_REQUESTS.
However there are a number of other limits that affect request queueing.
In particular there is a CONCORDIUM_NODE_GRPC2_MAX_CONCURRENT_REQUESTS_PER_CONNECTION limit which will limit how many concurrent requests are being processed, but instead of rejecting them it will queue them.
In general the node’s resource limits are meant to protect the node. You should not be using them as a job queue, you should queue transactions for submission on the client side instead of submitting 100k transactions.
Thanks for confirming the parameter(s). Yes, we are not using the node as a job queue; we have a setup for that based on Kafka. Depending on the needs, we can also have multiple nodes.
For submitting transactions, the request is deemed “completed” when the validation is done.
In practice this means whey you get the response, except in situations where you are having network problems (the request is deemed completed before the response is sent over the network (but after the response is constructed)).