Is there an ETA on the Python SDK?
There are no plans for a python SDK.
However you can get good mileage using grpc2 interface concordium-node/grpc2.md at main Ā· Concordium/concordium-node Ā· GitHub
This has very extensive schema files for responses concordium-grpc-api/types.proto at main Ā· Concordium/concordium-grpc-api Ā· GitHub concordium-grpc-api/service.proto at main Ā· Concordium/concordium-grpc-api Ā· GitHub which will likely lead to a good generated SDK if you follow steps here Protocol Buffer Basics: Python Ā |Ā Protocol Buffers Ā |Ā Google Developers
Itās not as good as a custom made SDK, but it is likely going to be good enough for queries.
1 Like
Wowā¦
At least 20 times faster than my current Concordium-client based implementation!
Questions:
- I canāt seem to decode the BlockHash value? I was able to decode an AccountAddress using
base58.b58encode_check(b'\x01' + result.address.value).decode(), but havenāt found a similar conversion that works for blockhashes. - Concordium-client (through the
rawcommand) outputs json. The output from grpc is in the form the classes that are created through protobuf. Is there a way to automatically have grpc output json as well? Am I missing a translation step somewhere? Or is this the step that would be performed in the SDK?
-
What do you mean by decode block hashes? What you get back is a byte array. If you want to convert it to a string you just have to
hexencode it. Is that what you mean? Hashes are just 32-byte arrays. -
Protobuf has a canonical encoding to JSON, see google.protobuf.json_format ā Protocol Buffers 4.21.1 documentation
This wonāt give you the same JSON format as V1 grpc (and concordium-client) gives at the moment, but if you donāt care that it is the same, just that it is JSON, this would be the recommended approach.
Iāve called GetBlockInfo with blockHash = BlockHash(value=bytes.fromhex("058bafd07ea9ed8d234a8832b6a04c906fa2f0358b892bcd53e92a445807c09d")) and got back:
blockInfo['hash']['value']
'BYuv0H6p7Y0jSogytqBMkG+i8DWLiSvNU+kqRFgHwJ0='
Hex encoding this leads to:
bytes(blockInfo['hash']['value'], encoding='ascii').hex() '42597576304836703759306a536f67797471424d6b472b693844574c6953764e552b6b7152466748774a303d', which is obviously not the same as the input. Encoding utf-8 leads to the same result.
On json, it does indeed convert to json, which is very good, however, it doesnāt automatically convert hashes to strings (and other conversions). For example, for GetAccountInfo, the resulting json has a key amount, which is dict with a key value, while v1 (or Concordium-client collapses the value key directly into amount).
BYuv0H6p7Y0jSogytqBMkG+i8DWLiSvNU+kqRFgHwJ0= is base64 encoding of the same byte array as 058bafd07ea9ed8d234a8832b6a04c906fa2f0358b892 is the hex encoding of.
I have not used the python bindings, but it looks like it uses base64 encoding for binary data.
That worked, thank you. And the other question on automatically converting values, Iām assuming I need to do this myself on a case by case basis?
I think so, but I am not familiar with the details of the python library. Perhaps it has some knobs to tweak JSON representation of objects with a single field.
Hereās a partial implementation I worked on:
def convertSingleValue(value):
if type(value) in [BlockHash, TransactionHash, StateHash, ModuleRef]:
return base64.b64decode(MessageToDict(value)['value']).hex()
if type(value) in [AccountAddress]:
return convertAccountAddress(value)
elif type(value) == Timestamp:
return dt.datetime.fromtimestamp(int(MessageToDict(value)['value']) / 1_000)
elif type(value) == AccountStakingInfo:
pass
elif type(value) in [int, bool, str, float]:
return value
elif type(value.value) == int:
if 'value' in MessageToDict(value):
return int(MessageToDict(value)['value'])
else:
return 0
def get_key_value_from_descriptor(name, the_list):
return name, getattr(the_list, name)
def convertAccountAddress(value):
return base58.b58encode_check(b'\x01' + value.value).decode()
def convertAmount(value):
return value.value
def convertCommissionRates(value):
result = {}
for descriptor in value.DESCRIPTOR.fields:
key, val = get_key_value_from_descriptor(descriptor.name, value)
result[key] = val.parts_per_hundred_thousand / 100_000
return result
def get_pool_info_for_pool(pool_id:int, block_hash: str):
result = {}
blockHash = BlockHash(value=bytes.fromhex(block_hash))
baker_id = BakerId(value=pool_id)
poolInfoRequest = PoolInfoRequest(
baker=baker_id,
block_hash=BlockHashInput(given=blockHash))
grpc_return_value = stub.GetPoolInfo(request=poolInfoRequest)
for descriptor in grpc_return_value.DESCRIPTOR.fields:
key, value = get_key_value_from_descriptor(descriptor.name, grpc_return_value)
if key not in ['pool_info', 'equity_pending_change', 'current_payday_info']:
result[key] = convertSingleValue(value)
elif key == 'pool_info':
pool_info_value = value
pool_info_dict = {}
for descriptor in pool_info_value.DESCRIPTOR.fields:
key1, value1 = get_key_value_from_descriptor(descriptor.name, pool_info_value)
if key1 == 'open_status':
pool_info_dict[key1] = OpenStatusEnum(value1).name
elif key1 == 'url':
pool_info_dict[key1] = convertSingleValue(value1)
elif key1 == 'commission_rates':
pool_info_dict[key1] = convertCommissionRates(value1)
result['pool_info'] = pool_info_dict
elif key == 'current_payday_info':
current_payday_value = value
current_payday_dict = {}
for descriptor in current_payday_value.DESCRIPTOR.fields:
key1, value1 = get_key_value_from_descriptor(descriptor.name, current_payday_value)
current_payday_dict[key1] = convertSingleValue(value1)
result['current_payday_info'] = current_payday_dict
return result
Which should return output similar to GetPoolStatus on concordium-client.
pool_info = get_pool_info_for_pool(72723, "affea3382993132bf8fd4f6b5c8548e015ca5b99b074a4f2df57d4878cfce829")
yields:
{
'baker': 72723,
'address': '3BFChzvx3783jGUKgHVCanFVxyDAn5xT3Y5NL5FKydVMuBa7Bm',
'equity_capital': 1097960678918,
'delegated_capital': 891973258115,
'delegated_capital_cap': 2195921357836,
'pool_info': {
'open_status': 'openForAll',
'url': 'https://concordium-explorer.nl/delegate-to-72723-for-free-access/',
'commission_rates': {'finalization': 1.0, 'baking': 0.1, 'transaction': 0.1}
},
'current_payday_info': {
'blocks_baked': 0,
'finalization_live': False,
'transaction_fees_earned': 0,
'effective_stake': 1988438144006,
'lottery_power': 0.00023535173260320042,
'baker_equity_capital': 1097068380081,
'delegated_capital': 891369763925
},
'all_pool_total_capital': 8653044522552950
}
Now, this isnāt bad, but I feel Iām essentially building a SDK? As there are many types and types are nested really deep. I havenāt found a way to do a recursive type conversion.
Iād be very interested to learn how the other SDKs approach this, as I would have to assume that they have all implemented similar features? For example, in the NodeJS SDK, thereās this typeTranslation.ts file, which implements a similar translation method. However, this method transBaker never gets called anywhere in this repo?
Do you think this is the right approach and I should just plough through, or is there an easier path?
In the Rust SDK we do essentially this. We build the conversions bottom up as it were. It is quite a bit of work but it is also straightforward work. And as a result we do get a much better user experience.
However is it a major problem for you that the values are nested like they are?
The nodeJS SDK is in the process of being migrated to V2 API at the moment.
Ideally, Iād want to attach a method convert to each generated type, that instructs how to convert this value. When calling a query and receiving a (compound) type, the MessageToDict method would internally call the convert method on each type it finds.
The (arbitrary) nesting does pose a problem for converting all types youād encounter traversing through the compound type, as I havenāt found a way to recursively walk through a protobuf Message. How is this done in the Rust SDK?
Alright, I think Iāve settled on an approach for this and yes, itās a lot of work.
As an example, this is my current implementation to decode AccountStakingInfo:
def convertAccountStakingInfo(self, message):
result = {}
which_one = message.WhichOneof("staking_info")
if which_one == "baker":
for descriptor in getattr(message, which_one).DESCRIPTOR.fields:
key, value = self.get_key_value_from_descriptor(descriptor, getattr(message, which_one))
if type(value) in [BakerId, AccountAddress, Amount, str, int, bool, float]:
result[key] = self.convertSingleValue(value)
elif type(value) == BakerPoolInfo:
result[key] = self.convertBakerPoolInfo(value)
elif type(value) == BakerInfo:
result[key] = self.convertBakerInfo(value)
elif type(value) == StakePendingChange:
result[key] = self.converPendingChange(value)
elif which_one == "delegator":
for descriptor in getattr(message, which_one).DESCRIPTOR.fields:
key, value = self.get_key_value_from_descriptor(descriptor, getattr(message, which_one))
if type(value) in [BakerId, AccountAddress, Amount, str, int, bool, float]:
result[key] = self.convertSingleValue(value)
elif type(value) == DelegationTarget:
result[key] = self.convertDelegationTarget(value)
elif type(value) == StakePendingChange:
result[key] = self.convertPendingChange(value)
return result
Whatās nice is that, from the specification, I know exactly which type to expect and using type(value) I can switch based on these types.
Where this approach falls down, though, is with repeated values.
This is my implementation for Release:
def convertRelease(self, message):
resulting_dict = {}
for descriptor in message.DESCRIPTOR.fields:
key, value = self.get_key_value_from_descriptor(descriptor, message)
if key == 'schedules':
schedule = []
for entry in value:
entry_dict = {}
for descriptor in entry.DESCRIPTOR.fields:
key, value = self.get_key_value_from_descriptor(descriptor, entry)
if key == 'transactions':
entry_dict[key] = self.convertList(value)
elif type(value) == Timestamp:
entry_dict[key] = self.convertSingleValue(value)
schedule.append(entry_dict)
resulting_dict[key] = schedule
elif type(value) == Amount:
resulting_dict[key] = self.convertSingleValue(value)
return resulting_dict
I canāt lookup the type from type(value), as when the key == transactions, the type(value) == <class 'google._upb._message.RepeatedCompositeContainer'>.
Is this something inherent to Protobuf?
Finally, is there a reference implementation for v2, such that I can check my implementation?
In particular, Iām looking for DelegationTarget? The documentation mentions a passive field of type Empty and a baker field of type BakerId. In my tests, however, for a passive delegator the baker field is empty, while for a delegator to a baker, the baker field has the baker id value.
In these cases, what would be the correct value for 'target': {...}?
Talking to myself hereā¦
I tried the Rust-SDK implementation, in the example folder v2_get_block_info.rs.
This outputs:
Last finalized QueryResponse {
block_hash: 40f0b310,
response: BlockInfo {
transactions_size: 664,
block_parent: 5427d0d8,
block_hash: 40f0b310,
finalized: true,
block_state_hash: 80758ded,
block_arrive_time: 2023-01-21T13:13:04.990Z,
block_receive_time: 2023-01-21T13:13:04.952Z,
transaction_count: 4,
transaction_energy_cost: Energy {
energy: 1980,
},
block_slot: Slot {
slot: 13552301,
},
block_last_finalized: 5427d0d8,
block_slot_time: 2023-01-21T13:13:04.750Z,
block_height: AbsoluteBlockHeight {
height: 5026231,
},
era_block_height: BlockHeight {
height: 334385,
},
genesis_index: GenesisIndex {
height: 4,
},
block_baker: Some(
BakerId {
id: AccountIndex {
index: 5,
},
},
),
},
}
Ie, a literal translation of the BlockInfo object from the .proto file. Compare this to what the client (I know, v1) currently outputs (different block):
{
"blockArriveTime": "2023-01-21T13:15:21.429621Z",
"blockBaker": 8,
"blockHash": "ae607f0d547f0f45fc65b7faef87cfb8804b9a9f1dff7bde5ca92f732aa6dfdd",
"blockHeight": 5026248,
"blockLastFinalized": "c866915b953d1d94e8fbfa3e550fa917c6858555eeffc294e882815cad821f19",
"blockParent": "c866915b953d1d94e8fbfa3e550fa917c6858555eeffc294e882815cad821f19",
"blockReceiveTime": "2023-01-21T13:15:21.402247Z",
"blockSlot": 13552847,
"blockSlotTime": "2023-01-21T13:15:21.25Z",
"blockStateHash": "24eb3e4f139a1500f6c90e28c78756f3d1756b8bbe960d606353168d2d684c61",
"eraBlockHeight": 334402,
"finalized": true,
"genesisIndex": 4,
"transactionCount": 0,
"transactionEnergyCost": 0,
"transactionsSize": 0
}
What am I supposed to target as output? Underscore names of CamelCase? Also (maybe more of a Rust questionā¦) why are the block hashes limited to 8 characters?
Related to Rust. So what you showed there was Debug output that is meant for, well, debugging. Not the JSON output. The JSON output is as you showed in the followup. The Rust type into which we convert still has a JSON conversion instance (but that is mainly for legacy reasons, we donāt use it elsewhere).
So I am not entirely sure what you are suggesting. The reason the JSON instance is the way it is is because it used to be like that in V1 GRPC version.
concordium-rust-sdk/src/v2/conversions.rs at main Ā· Concordium/concordium-rust-sdk Ā· GitHub would be closest to what I would call āreference implementationā.
Do you mean an implementation which is clear about what the response format is, or what do you expect from a reference implementation?
I would expect, for example, the Rust SDK and nodeJS SDK to have the same output for the same commands? If Iām building my own mini Python SDK, Iād like to stick as close as possible to this same output, if that makes sense?
In general I think Iām just confused as to what the output of the official SDKs is. My assumption now is that, for example for GetBlockInfo, you return a BlockInfo type, which contains BlockHash (and others), but this BlockHash is not decoded back to hex at this time (compare and contrast to the output from the Concordium-client).
On a related note: I tried to automatically create Pydantic classes from the protobuf files, but this fails with a mention that the .proto files were created with a version <3.20.1? Does that ring true to you? if so, can you/will you at some point re create them with a more recent version? Iām now writing my Pedantic classes by hand, which is pretty tedious.
I donāt know what you mean here. What commands? THere is a type, defined in rust, called BlockInfo or some such. That is defined as the return type of get_block_info or some such. This type is a refined version (as in, does more validation) of the type defined in the .proto file. That is necessary since protobuf is much more limited than Rust.
So the .proto files were created by hand, not by any tool. They should work with protobuf-compiler 3.15 and up. What version do you have installed? What is it that goes wrong? Iām not familiar with pydantic.
A command would be āgetBlockInfoā , but I guess youāve answered my question by stating that youāve defined additional types in Rust that perform validation. Thatās exactly the goal of Pydantic models.
Ok, I finally found an example of what Iām trying to convey.
Below is a snippet from GetAccountInfo.html from the NodeJS Web repo:
<script type="module" >
const client = new concordiumSDK.JsonRpcClient(new concordiumSDK.HttpProvider("http://localhost:9095"));
const update = function () {
client.getAccountInfo(new concordiumSDK.AccountAddress(address.value)).then((res) => {
console.log(res);
document.getElementById('balance').innerHTML = "Balance: " + res.accountAmount.toString() + " microCCD";
document.getElementById('showAddress').innerHTML = address.value;
});
};
This SDK also uses the .proto files and hence, AccountInfo has a property amount that is of type Amount, which has a property value which contains the microCCD value.
Observations:
- the property on the result of
getAccountInfois calledaccountAmount(and notamount, as the.protofile would indicate). - From the above snippet it is clear to me that the SDK converts the
Accounttype it has received fromgetAccountInfointo anint(or similar), otherwise we would see a something like:
"Balance: " + res.accountAmount.value.toString() + " microCCD";
What Iām trying to say here, is that this SDK directly returns an amount as int and NOT a property with type Amount.
To give another example, suppose youāre requesting (in this SDK) getBlockInfo. The return type has a property called hash, which is of type BlockHash, which has a property value which is in bytes.
My assumption is (this example isnāt listed in the repo) that this SDK will output from client.getBlockInfo as result that has a property that is the hex output and NOT the bytes output, that is the SDK converts the BlockHash type from bytes to hex and returns the hex output (of type str) and not the BlockHash type.
Is my question somewhat clearer now?