[FIX] [RPC] Align getrawtransaction with Bitcoin Core#985
Conversation
044b3b7 to
6099cc1
Compare
|
This PR is in draft because it depends on #821. That PR introduces typos for spell checking, allowing more flexibility when using randomly generated addresses without triggering spelling errors |
9547e4a to
c9ac410
Compare
c9ac410 to
22e29a2
Compare
|
22e29a2 I rebased and converted the added integration test to the pytest format. |
| #[command(name = "gettransaction")] | ||
| GetTransaction { txid: Txid, verbose: Option<bool> }, | ||
| #[command(name = "getrawtransaction")] | ||
| GetRawTransaction { txid: Txid, verbose: Option<u32> }, |
There was a problem hiding this comment.
I dont think that Bitcoin Core will appear with 4 294 967 295 levels of verbosity
There was a problem hiding this comment.
I agree, but u32 is the default for numeric verbosity that we use.
There was a problem hiding this comment.
u32 occupies too much space for something simple as a verbosity level.
Also: #1076
There was a problem hiding this comment.
I replaced the u32 with a u8.
| match verbosity { | ||
| 0 => serde_json::to_value(serialize_hex(&tx.tx)) | ||
| .map_err(|e| JsonRpcError::Decode(e.to_string())), | ||
| 1 => serde_json::to_value(self.make_raw_transaction(tx)) | ||
| .map_err(|e| JsonRpcError::Decode(e.to_string())), | ||
| _ => return Err(JsonRpcError::InvalidVerbosityLevel), | ||
| } | ||
| } |
There was a problem hiding this comment.
Return a structured type instead of a generic
|
|
||
| /// The information returned by a get_raw_tx | ||
| #[derive(Deserialize, Serialize)] | ||
| pub struct RawTxJson { |
There was a problem hiding this comment.
Because the GetRawTransaction of corepc is not good, the coinbase field of the input is missing, and in addition, old fields are present.
22e29a2 to
d67326e
Compare
|
nit: the bang ( |
| if verbosity > 1 { | ||
| return Err(JsonRpcError::InvalidVerbosityLevel); | ||
| } |
There was a problem hiding this comment.
nit: Since there's a wildcard inside the match below, this check is useless.
There was a problem hiding this comment.
Yeah, I didn’t want the verbosity check to be after getting the transaction, but it doesn’t really matter, since it would only save a few milliseconds in the process. So I removed this if; the validation is in the match now.
| pub enum GetRawTransactionRes { | ||
| Zero(String), | ||
|
|
||
| One(Box<RawTxJson>), | ||
| } |
There was a problem hiding this comment.
nit:
| pub enum GetRawTransactionRes { | |
| Zero(String), | |
| One(Box<RawTxJson>), | |
| } | |
| pub enum GetRawTransactionRes { | |
| Zero(String), | |
| One(Box<RawTxJson>), | |
| } |
Also, you Box-ed it because it is too big?
There was a problem hiding this comment.
Yes, we are adopting Box for the response enums because the verbosity objects tend to be very large.
| } | ||
|
|
||
| /// The information returned by a get_raw_tx | ||
| #[derive(Debug, Deserialize, Serialize)] |
There was a problem hiding this comment.
| #[derive(Debug, Deserialize, Serialize)] | |
| #[derive(Debug, Deserialize, Serialize)] | |
| /// The information returned by a get_raw_tx |
d67326e to
7331f65
Compare
|
Needs rebase |
7331f65 to
d0b8750
Compare
|
d0b8750 I’ve rebased |
| #[command(name = "gettransaction")] | ||
| GetTransaction { txid: Txid, verbose: Option<bool> }, | ||
| #[command(name = "getrawtransaction")] | ||
| GetRawTransaction { txid: Txid, verbose: Option<u8> }, |
There was a problem hiding this comment.
verbosity (in other places as well).
| let raw_tx = self.make_raw_transaction(tx)?; | ||
| serde_json::to_value(raw_tx).map_err(|e| JsonRpcError::Decode(e.to_string())) | ||
| } | ||
| _ => Err(JsonRpcError::InvalidVerbosityLevel), |
There was a problem hiding this comment.
I think core always return the maximum known level if the number isn't defined — if I pass 3, it returns the same as 1
There was a problem hiding this comment.
No, in getrawtransaction, Bitcoin Core returns 0 as the default value, according to the documentation.
https://bitcoincore.org/en/doc/30.0.0/rpc/rawtransactions/getrawtransaction/
| } | ||
|
|
||
| #[derive(Debug, Deserialize, Serialize)] | ||
| /// The information returned by a get_raw_tx |
There was a problem hiding this comment.
Doesn't corepc have those types?
There was a problem hiding this comment.
yes, but the GetRawTransaction of corepc is not good, the coinbase field of the input is missing, and in addition, old fields are present.
There was a problem hiding this comment.
Perhaps one an issue and leave a TODO here?
There was a problem hiding this comment.
I checked the newer versions of corepc, and the getrawtransaction structure has been improved. So I updated to the latest version and used its struct here.
| /// The weight of this transaction, as defined by the segwit soft-fork | ||
| pub weight: u32, | ||
|
|
||
| /// This transaction's version. The current bigger version is 2 |
There was a problem hiding this comment.
Actually, three (TRUC transactions)
| #[derive(Debug, Deserialize, Serialize)] | ||
| /// The locking script inside a txout | ||
| pub struct ScriptPubKeyJson { | ||
| /// A ASM representation for this script |
| .witness | ||
| .iter() | ||
| .map(|w| w.to_hex_string(bitcoin::hex::Case::Upper)) | ||
| .map(|w| w.to_hex_string(bitcoin::hex::Case::Lower)) |
|
|
||
| /// TODO: This function is not compliant with Bitcoin Core. | ||
| /// See: <https://github.com/getfloresta/Floresta/issues/987> | ||
| pub(super) fn get_script_type_descriptor(script: &Script, address: &Option<Address>) -> String { |
There was a problem hiding this comment.
These methods could be an extension trait for Address or Script
There was a problem hiding this comment.
Yes, I’m planning to work on that in issue #1108, so this will be a separate PR to add extensions for transaction items, since some parts are shared between floresta-node and floresta-electrum.
| block = self.bitcoind.rpc.get_block(bestblockhash) | ||
| txids.append(block["tx"][0]) # Get the coinbase transaction txid | ||
|
|
||
| self.utreexod = utreexod_node |
There was a problem hiding this comment.
In this case, Utreexo is only used here to provide transaction proofs to Floresta, so it basically just connects to other peers and performs the IBD. That’s why it is only called in this lower part.
There was a problem hiding this comment.
Right, I removed it.
d4c3dc2 to
b99a917
Compare
|
b99a917 Added a helper function to compare fields, which helps in cases where we need to compare responses against Bitcoin Core. Also rebased the branch. |
a09ec26 to
20b0dad
Compare
|
@moisesPompilio CI broke |
20b0dad to
a0cf65d
Compare
| with pytest.raises(HTTPError): | ||
| self.florestad.rpc.get_raw_transaction("nonexistingtxid") | ||
|
|
||
| with pytest.raises(HTTPError): | ||
| self.florestad.rpc.get_raw_transaction("nonexistingtxid", verbose=0) | ||
|
|
||
| with pytest.raises(HTTPError): | ||
| self.florestad.rpc.get_raw_transaction("nonexistingtxid", verbose=1) |
There was a problem hiding this comment.
Now we have tooling for the framework to precisely assert against rpc errors, dont you wanna use it here ?
|
|
||
| txids = [] | ||
| value = 6.4242521 | ||
| for address in [ |
There was a problem hiding this comment.
I see other tests that can use a chain with transactions, perhaps we could extract this into a shared helper ? Same thing we did with the wait_until wrapper
There was a problem hiding this comment.
Hmm, I think it's better to leave this for later. I'd like this helper function to be implemented using Floresta's wallet functionality rather than Bitcoin Core's
| raise TimeoutError(f"{error_msg} after {timeout} seconds") | ||
|
|
||
|
|
||
| def compare_fields(candidate, reference, ignore_fields=None, float_tol=1e-8): |
| for key, ref_value in reference.items(): | ||
| if key in ignore_fields: | ||
| continue | ||
| assert key in candidate, f"Missing key in candidate: {key}" | ||
| compare_fields(candidate[key], ref_value, ignore_fields=ignore_fields) |
There was a problem hiding this comment.
THis method doesnt need to be recursive but okay
There was a problem hiding this comment.
Actually, it is necessary. We have cases with dicts nested inside other dicts, as well as dicts inside lists that are themselves nested within dicts. That's why the recursion is needed. A direct comparison would not work for those cases.
| return | ||
|
|
||
| # scalar | ||
| assert candidate == reference |
There was a problem hiding this comment.
This have the "failfast" behavior right ? itll stop comparing at the first difference, dont we want to know all fields that differ (in a single run)?
There was a problem hiding this comment.
Yes, this is fail-fast because it relies on assertions. As soon as a comparison fails, the entire comparison chain stops.
In this case, collecting all differences is not really feasible because the function is generic and does not know the structure of the data beforehand. For example, a getrawtransaction response contains a top-level dict, a vin field that is a list, and each item in that list is another dict. Since the structure can vary depending on the RPC response being compared, the function recursively traverses the data and fails as soon as it encounters the first mismatch.
| for height in range(initial_block_coinbase_wallet, best_block_height): | ||
| block_hash = self.bitcoind.rpc.get_blockhash(height) | ||
|
|
||
| block = self.bitcoind.rpc.get_block(block_hash, verbosity=1) | ||
| coinbase_tx = block["tx"][0] # Get the coinbase transaction txid | ||
|
|
||
| self.compare_getrawtransaction(coinbase_tx) |
There was a problem hiding this comment.
You are checking coinbases twice ?
| - `InvalidVerbosityLevel` - Verbosity parameter is not 0 or 1 | ||
| - `MissingParameter` - The txid parameter was not provided | ||
| - `InvalidParameterType` - The txid is not a valid hex string | ||
| - `Decode` - Error during transaction decoding or serialization |
There was a problem hiding this comment.
NIT: I would like to see a note about the requirement of the wallet to track the transaction or we will not find it
0396af7 to
b2e9a69
Compare
…osity to numeric type The RPC client was incorrectly calling 'gettransaction' (which doesn't exist on the server) instead of 'getrawtransaction'. Additionally, the verbosity parameter handling was inconsistent. While the method signature correctly accepted Option<u32>, it needed better standardization to match Bitcoin Core's RPC specification.
Bump `corepc-types` to v0.14.0, bringing various updates and fixes to Bitcoin Core RPC type definitions and improving compatibility with newer RPC fields and responses.
…format This commit aligns the getrawtransaction RPC response with Bitcoin Core's format and behavior. The following changes were made: RawTx struct: - Made blockhash, confirmations, blocktime, and time optional fields They are now only serialized when present, avoiding null values - Fixed txid encoding that was previously inverted TxOut struct: - Changed value from u64 (sats) to f64 (BTC) for proper denomination - Made address field optional, only serialized when script can be converted to a valid address TxIn struct: - Added coinbase field that only appears in coinbase transactions, containing the hex-encoded script from scriptSig - Coinbase inputs have: coinbase, sequence, witness (optional) - Regular inputs have: txid, vout, script_sig, sequence, witness (optional) Script handling: - Improved ASM (assembly) format conversion to strictly follow Bitcoin Core rules and formatting standards TxOut types: - Updated to include all script types that Bitcoin Core supports Note: The `desc` field was not implemented due to complex matching rules. This will be added in a future PR.
Add a helper for recursively comparing RPC responses while ignoring selected fields. Use it in the `getblockchaininfo`, `getblockheader`, `gettxout` and `getblock` tests to simplify response validation between Floresta and Bitcoin Core.
Merge `noraise_raw_request` and `noraise_raw_data_request` into a single `noraise_raw_request` helper. The new implementation automatically handles dictionary payloads using the previous JSON-RPC behavior and falls back to the raw data request path for non-dictionary payloads. Move commonly used RPC helper functions into `BaseRPC` so they are available to all nodes without requiring additional imports. Also remove redundant helper methods from the RPC base implementation.
Add comprehensive integration test comparing Floresta's `getrawtransaction` RPC output with Bitcoin Core for verbosity levels 0 and 1, covering: - Regular transactions (`P2PKH`, `P2WPKH`, `P2SH`, `P2TR types`) - Coinbase transactions - Input/output field validation - Optional field handling New BaseRPC methods: - `get_raw_transaction()` New BitcoinRPC methods: - `create_wallet()` - `get_new_address()` - `generate_block_to_wallet()` - `send_to_address()`
b2e9a69 to
eb13fe8
Compare
|
eb13fe8 I rebased |
Description and Notes
Brings Floresta’s
getrawtransactionRPC closer to Bitcoin Core compatibility.Fixes the RPC method call so the client correctly uses
getrawtransactioninstead of the incorrectgettransaction, and standardizesverbosityhandling to match Bitcoin Core’s numeric behavior.The response format was aligned with Bitcoin Core as well:
RawTxfields such asblockhash,confirmations,blocktime, andtimeare now optional and only serialized when present.txidencoding was corrected.TxOut.valuenow usesf64in BTC instead ofu64in sats, matching Bitcoin Core’s denomination.TxOut.addressis now optional and only included when the script can be converted into a valid address.TxInnow handles coinbase inputs properly, including thecoinbasefield only when applicable.TxOutscript types were expanded to cover the script types supported by Bitcoin Core.Note: the
descfield was intentionally left out, since matching Bitcoin Core’s descriptor logic requires additional investigation and a more complex implementation. That can be handled in a future PR.How to verify the changes you have done?
Compare the output of
getrawtransactionbetween Floresta and Bitcoin Core for bothverbosity = 0andverbosity = 1.The integration test added here already covers this behavior and is the best reference for validation.
P2PKH,P2WPKH,P2SH, andP2TR.