Python SDK ETA (extra characters to make the title longer than 15 characters...)

Is there an ETA on the Python SDK?

There are no plans for a python SDK.

However you can get good mileage using grpc2 interface concordium-node/grpc2.md at main · Concordium/concordium-node · GitHub

This has very extensive schema files for responses concordium-grpc-api/types.proto at main · Concordium/concordium-grpc-api · GitHub concordium-grpc-api/service.proto at main · Concordium/concordium-grpc-api · GitHub which will likely lead to a good generated SDK if you follow steps here Protocol Buffer Basics: Python  |  Protocol Buffers  |  Google Developers

It’s not as good as a custom made SDK, but it is likely going to be good enough for queries.

1 Like

Wow…

At least 20 times faster than my current Concordium-client based implementation!

Questions:

  1. I can’t seem to decode the BlockHash value? I was able to decode an AccountAddress using base58.b58encode_check(b'\x01' + result.address.value).decode(), but haven’t found a similar conversion that works for blockhashes.
  2. Concordium-client (through the raw command) outputs json. The output from grpc is in the form the classes that are created through protobuf. Is there a way to automatically have grpc output json as well? Am I missing a translation step somewhere? Or is this the step that would be performed in the SDK?
  1. What do you mean by decode block hashes? What you get back is a byte array. If you want to convert it to a string you just have to hex encode it. Is that what you mean? Hashes are just 32-byte arrays.

  2. Protobuf has a canonical encoding to JSON, see google.protobuf.json_format — Protocol Buffers 4.21.1 documentation

    This won’t give you the same JSON format as V1 grpc (and concordium-client) gives at the moment, but if you don’t care that it is the same, just that it is JSON, this would be the recommended approach.

I’ve called GetBlockInfo with blockHash = BlockHash(value=bytes.fromhex("058bafd07ea9ed8d234a8832b6a04c906fa2f0358b892bcd53e92a445807c09d")) and got back:

blockInfo['hash']['value']
'BYuv0H6p7Y0jSogytqBMkG+i8DWLiSvNU+kqRFgHwJ0='

Hex encoding this leads to:
bytes(blockInfo['hash']['value'], encoding='ascii').hex() '42597576304836703759306a536f67797471424d6b472b693844574c6953764e552b6b7152466748774a303d', which is obviously not the same as the input. Encoding utf-8 leads to the same result.

On json, it does indeed convert to json, which is very good, however, it doesn’t automatically convert hashes to strings (and other conversions). For example, for GetAccountInfo, the resulting json has a key amount, which is dict with a key value, while v1 (or Concordium-client collapses the value key directly into amount).

BYuv0H6p7Y0jSogytqBMkG+i8DWLiSvNU+kqRFgHwJ0= is base64 encoding of the same byte array as 058bafd07ea9ed8d234a8832b6a04c906fa2f0358b892 is the hex encoding of.

I have not used the python bindings, but it looks like it uses base64 encoding for binary data.

That worked, thank you. And the other question on automatically converting values, I’m assuming I need to do this myself on a case by case basis?

I think so, but I am not familiar with the details of the python library. Perhaps it has some knobs to tweak JSON representation of objects with a single field.

Here’s a partial implementation I worked on:

def convertSingleValue(value):
    if type(value) in [BlockHash, TransactionHash, StateHash, ModuleRef]:
        return base64.b64decode(MessageToDict(value)['value']).hex()
    if type(value) in [AccountAddress]:
        return convertAccountAddress(value)
    elif type(value) == Timestamp:
        return dt.datetime.fromtimestamp(int(MessageToDict(value)['value']) / 1_000)
    elif type(value) == AccountStakingInfo:
        pass
    elif type(value) in [int, bool, str, float]:
        return value
    elif type(value.value) == int:
        if 'value' in MessageToDict(value):
            return int(MessageToDict(value)['value'])    
        else:
            return 0

def get_key_value_from_descriptor(name, the_list):
    return name, getattr(the_list, name)

def convertAccountAddress(value):
    return base58.b58encode_check(b'\x01' + value.value).decode()

def convertAmount(value):
    return value.value

def convertCommissionRates(value):
    result = {}
    for descriptor in value.DESCRIPTOR.fields:
        key, val = get_key_value_from_descriptor(descriptor.name, value)
        result[key] = val.parts_per_hundred_thousand / 100_000
    return result

def get_pool_info_for_pool(pool_id:int, block_hash: str):
    result = {}
    blockHash           = BlockHash(value=bytes.fromhex(block_hash))
    baker_id            = BakerId(value=pool_id)
    poolInfoRequest     = PoolInfoRequest(
                            baker=baker_id, 
                            block_hash=BlockHashInput(given=blockHash))
    grpc_return_value   = stub.GetPoolInfo(request=poolInfoRequest)
    
    for descriptor in grpc_return_value.DESCRIPTOR.fields:
        key, value = get_key_value_from_descriptor(descriptor.name, grpc_return_value)

        if key not in ['pool_info', 'equity_pending_change', 'current_payday_info']:
            result[key] = convertSingleValue(value)
        elif key == 'pool_info':
            pool_info_value = value
            pool_info_dict = {}

            for descriptor in pool_info_value.DESCRIPTOR.fields:
                key1, value1 = get_key_value_from_descriptor(descriptor.name, pool_info_value)
                if key1 == 'open_status':
                    pool_info_dict[key1] = OpenStatusEnum(value1).name
                elif key1 == 'url':
                    pool_info_dict[key1] = convertSingleValue(value1)
                elif key1 == 'commission_rates':
                    pool_info_dict[key1] = convertCommissionRates(value1)
            result['pool_info'] = pool_info_dict

        elif key == 'current_payday_info':
            current_payday_value = value
            current_payday_dict = {}
            for descriptor in current_payday_value.DESCRIPTOR.fields:
                key1, value1 = get_key_value_from_descriptor(descriptor.name, current_payday_value)
                current_payday_dict[key1] = convertSingleValue(value1)
            result['current_payday_info'] = current_payday_dict
            
    return result

Which should return output similar to GetPoolStatus on concordium-client.

pool_info = get_pool_info_for_pool(72723, "affea3382993132bf8fd4f6b5c8548e015ca5b99b074a4f2df57d4878cfce829")

yields:

{
    'baker': 72723,
    'address': '3BFChzvx3783jGUKgHVCanFVxyDAn5xT3Y5NL5FKydVMuBa7Bm',
    'equity_capital': 1097960678918,
    'delegated_capital': 891973258115,
    'delegated_capital_cap': 2195921357836,
    'pool_info': {
        'open_status': 'openForAll',
        'url': 'https://concordium-explorer.nl/delegate-to-72723-for-free-access/',
        'commission_rates': {'finalization': 1.0, 'baking': 0.1, 'transaction': 0.1}
    },
    'current_payday_info': {
        'blocks_baked': 0,
        'finalization_live': False,
        'transaction_fees_earned': 0,
        'effective_stake': 1988438144006,
        'lottery_power': 0.00023535173260320042,
        'baker_equity_capital': 1097068380081,
        'delegated_capital': 891369763925
    },
    'all_pool_total_capital': 8653044522552950
}

Now, this isn’t bad, but I feel I’m essentially building a SDK? As there are many types and types are nested really deep. I haven’t found a way to do a recursive type conversion.

I’d be very interested to learn how the other SDKs approach this, as I would have to assume that they have all implemented similar features? For example, in the NodeJS SDK, there’s this typeTranslation.ts file, which implements a similar translation method. However, this method transBaker never gets called anywhere in this repo?

Do you think this is the right approach and I should just plough through, or is there an easier path?

In the Rust SDK we do essentially this. We build the conversions bottom up as it were. It is quite a bit of work but it is also straightforward work. And as a result we do get a much better user experience.

However is it a major problem for you that the values are nested like they are?

The nodeJS SDK is in the process of being migrated to V2 API at the moment.

Ideally, I’d want to attach a method convert to each generated type, that instructs how to convert this value. When calling a query and receiving a (compound) type, the MessageToDict method would internally call the convert method on each type it finds.

The (arbitrary) nesting does pose a problem for converting all types you’d encounter traversing through the compound type, as I haven’t found a way to recursively walk through a protobuf Message. How is this done in the Rust SDK?

Alright, I think I’ve settled on an approach for this and yes, it’s a lot of work.

As an example, this is my current implementation to decode AccountStakingInfo:

def convertAccountStakingInfo(self, message):
        result = {}
        which_one = message.WhichOneof("staking_info")
        if which_one == "baker":
            for descriptor in getattr(message, which_one).DESCRIPTOR.fields:
                key, value = self.get_key_value_from_descriptor(descriptor, getattr(message, which_one))
                if type(value) in [BakerId, AccountAddress, Amount, str, int, bool, float]:
                    result[key] = self.convertSingleValue(value)
                
                elif type(value) == BakerPoolInfo:
                    result[key] = self.convertBakerPoolInfo(value)

                elif type(value) == BakerInfo:
                    result[key] = self.convertBakerInfo(value)

                elif type(value) == StakePendingChange:
                    result[key] = self.converPendingChange(value)

        elif which_one == "delegator":
            for descriptor in getattr(message, which_one).DESCRIPTOR.fields:
                key, value = self.get_key_value_from_descriptor(descriptor, getattr(message, which_one))
                if type(value) in [BakerId, AccountAddress, Amount, str, int, bool, float]:
                    result[key] = self.convertSingleValue(value)
                
                elif type(value) == DelegationTarget:
                    result[key] = self.convertDelegationTarget(value)

                
                elif type(value) == StakePendingChange:
                    result[key] = self.convertPendingChange(value)
        
        return result

What’s nice is that, from the specification, I know exactly which type to expect and using type(value) I can switch based on these types.

Where this approach falls down, though, is with repeated values.

This is my implementation for Release:

def convertRelease(self, message):
        resulting_dict = {}
        
        for descriptor in message.DESCRIPTOR.fields:
            key, value = self.get_key_value_from_descriptor(descriptor, message)
            
            if key == 'schedules':
                schedule = []
                for entry in value:
                    entry_dict = {}
                    for descriptor in entry.DESCRIPTOR.fields:
                        key, value = self.get_key_value_from_descriptor(descriptor, entry)
                        if key == 'transactions':
                            entry_dict[key] = self.convertList(value)
                        elif type(value) == Timestamp:
                            entry_dict[key] = self.convertSingleValue(value)        
                    schedule.append(entry_dict)
                resulting_dict[key] = schedule
            elif type(value) == Amount:
                resulting_dict[key] = self.convertSingleValue(value) 
        return resulting_dict

I can’t lookup the type from type(value), as when the key == transactions, the type(value) == <class 'google._upb._message.RepeatedCompositeContainer'>.

Is this something inherent to Protobuf?

Finally, is there a reference implementation for v2, such that I can check my implementation?

In particular, I’m looking for DelegationTarget? The documentation mentions a passive field of type Empty and a baker field of type BakerId. In my tests, however, for a passive delegator the baker field is empty, while for a delegator to a baker, the baker field has the baker id value.
In these cases, what would be the correct value for 'target': {...}?

Talking to myself here…

I tried the Rust-SDK implementation, in the example folder v2_get_block_info.rs.

This outputs:

Last finalized QueryResponse {
    block_hash: 40f0b310,
    response: BlockInfo {
        transactions_size: 664,
        block_parent: 5427d0d8,
        block_hash: 40f0b310,
        finalized: true,
        block_state_hash: 80758ded,
        block_arrive_time: 2023-01-21T13:13:04.990Z,
        block_receive_time: 2023-01-21T13:13:04.952Z,
        transaction_count: 4,
        transaction_energy_cost: Energy {
            energy: 1980,
        },
        block_slot: Slot {
            slot: 13552301,
        },
        block_last_finalized: 5427d0d8,
        block_slot_time: 2023-01-21T13:13:04.750Z,
        block_height: AbsoluteBlockHeight {
            height: 5026231,
        },
        era_block_height: BlockHeight {
            height: 334385,
        },
        genesis_index: GenesisIndex {
            height: 4,
        },
        block_baker: Some(
            BakerId {
                id: AccountIndex {
                    index: 5,
                },
            },
        ),
    },
}

Ie, a literal translation of the BlockInfo object from the .proto file. Compare this to what the client (I know, v1) currently outputs (different block):

{
    "blockArriveTime": "2023-01-21T13:15:21.429621Z",
    "blockBaker": 8,
    "blockHash": "ae607f0d547f0f45fc65b7faef87cfb8804b9a9f1dff7bde5ca92f732aa6dfdd",
    "blockHeight": 5026248,
    "blockLastFinalized": "c866915b953d1d94e8fbfa3e550fa917c6858555eeffc294e882815cad821f19",
    "blockParent": "c866915b953d1d94e8fbfa3e550fa917c6858555eeffc294e882815cad821f19",
    "blockReceiveTime": "2023-01-21T13:15:21.402247Z",
    "blockSlot": 13552847,
    "blockSlotTime": "2023-01-21T13:15:21.25Z",
    "blockStateHash": "24eb3e4f139a1500f6c90e28c78756f3d1756b8bbe960d606353168d2d684c61",
    "eraBlockHeight": 334402,
    "finalized": true,
    "genesisIndex": 4,
    "transactionCount": 0,
    "transactionEnergyCost": 0,
    "transactionsSize": 0
}

What am I supposed to target as output? Underscore names of CamelCase? Also (maybe more of a Rust question…) why are the block hashes limited to 8 characters?

Related to Rust. So what you showed there was Debug output that is meant for, well, debugging. Not the JSON output. The JSON output is as you showed in the followup. The Rust type into which we convert still has a JSON conversion instance (but that is mainly for legacy reasons, we don’t use it elsewhere).

So I am not entirely sure what you are suggesting. The reason the JSON instance is the way it is is because it used to be like that in V1 GRPC version.

concordium-rust-sdk/conversions.rs at main · Concordium/concordium-rust-sdk · GitHub would be closest to what I would call “reference implementation”.

Do you mean an implementation which is clear about what the response format is, or what do you expect from a reference implementation?

I would expect, for example, the Rust SDK and nodeJS SDK to have the same output for the same commands? If I’m building my own mini Python SDK, I’d like to stick as close as possible to this same output, if that makes sense?

In general I think I’m just confused as to what the output of the official SDKs is. My assumption now is that, for example for GetBlockInfo, you return a BlockInfo type, which contains BlockHash (and others), but this BlockHash is not decoded back to hex at this time (compare and contrast to the output from the Concordium-client).

On a related note: I tried to automatically create Pydantic classes from the protobuf files, but this fails with a mention that the .proto files were created with a version <3.20.1? Does that ring true to you? if so, can you/will you at some point re create them with a more recent version? I’m now writing my Pedantic classes by hand, which is pretty tedious.

I don’t know what you mean here. What commands? THere is a type, defined in rust, called BlockInfo or some such. That is defined as the return type of get_block_info or some such. This type is a refined version (as in, does more validation) of the type defined in the .proto file. That is necessary since protobuf is much more limited than Rust.

So the .proto files were created by hand, not by any tool. They should work with protobuf-compiler 3.15 and up. What version do you have installed? What is it that goes wrong? I’m not familiar with pydantic.

A command would be ‘getBlockInfo’ , but I guess you’ve answered my question by stating that you’ve defined additional types in Rust that perform validation. That’s exactly the goal of Pydantic models.

Ok, I finally found an example of what I’m trying to convey.

Below is a snippet from GetAccountInfo.html from the NodeJS Web repo:

 <script  type="module" >
         const client = new concordiumSDK.JsonRpcClient(new concordiumSDK.HttpProvider("http://localhost:9095"));
         const update = function () {
             client.getAccountInfo(new concordiumSDK.AccountAddress(address.value)).then((res) => {
                 console.log(res);
                 document.getElementById('balance').innerHTML = "Balance: " + res.accountAmount.toString() + " microCCD";
                 document.getElementById('showAddress').innerHTML = address.value;
             });
         };

This SDK also uses the .proto files and hence, AccountInfo has a property amount that is of type Amount, which has a property value which contains the microCCD value.

Observations:

  1. the property on the result of getAccountInfo is called accountAmount (and not amount, as the .proto file would indicate).
  2. From the above snippet it is clear to me that the SDK converts the Account type it has received from getAccountInfo into an int (or similar), otherwise we would see a something like:
 "Balance: " + res.accountAmount.value.toString() + " microCCD";

What I’m trying to say here, is that this SDK directly returns an amount as int and NOT a property with type Amount.

To give another example, suppose you’re requesting (in this SDK) getBlockInfo. The return type has a property called hash, which is of type BlockHash, which has a property value which is in bytes.
My assumption is (this example isn’t listed in the repo) that this SDK will output from client.getBlockInfo as result that has a property that is the hex output and NOT the bytes output, that is the SDK converts the BlockHash type from bytes to hex and returns the hex output (of type str) and not the BlockHash type.

Is my question somewhat clearer now?