Introduces five new JSON-RPC methods that enable efficient queries for raw chain history data:
eth_queryBlockseth_queryTransactionseth_queryLogseth_queryTraceseth_queryTransfersPlease reference the draft PR for our motivation, full specification, rationale, and usage guidelines.
Good work!
Two important corner cases I hope will be addressed, as I have found out this being very annoying to deal with in the past, causing bad developer experience or “headache”
Distinguishing between incomplete node data and empty replies. Especially important for Monad. The JSON-RPC should return an error instead of no data found response if the node does not have the data because of its internal purging rules. Example: ask events in Monad block 1. Because nodes purge this data, we should could get reply “data not available on this node” instead of no events. Because Monad RPCs have various purge and censorship rules, it is otherwise impossible to know whether the data exists or this particular RPC just doesn’t have it.
Pagination / limit within the block: If th pagination limit gets interrupted in the middle a block, we should be somehow know about this and be able to resume gracefully. Example: block 1 has 100 events, block 2 has 100 events. Our limit is 150. What happens on resume? How do we know we could only half of the events of block 2.
Thanks for the feedback, here are my responses.
The latest version of the proposal better specifies this behavior (link). It introduces the concept of a server / implementation availability window for each method, which the server must compare against each request, and respond with an error if the requested block range falls outside of the window.
We also added a section that better specifies the “Block-aligned pagination” behavior (link). TL;DR - the server is allowed to return greater than the user-requested number of events to ensure that the response is block-aligned. We also renamed limit to target, which better describes how it works. To address your scenario directly, the server would finish collecting data from block 2 and then return 200 events.
Thanks @typedarray ,
This is definitely amazing work. To better understand how the things work I want to ask the following:
{order: "desc", fromBlock: "latest"} with no toBlock scans to block 0. But any range outside the availability window must return -32001. So on a node that doesn’t keep full history, that query always fails.Did I get that right, or am I missing something? If it is an issue, maybe "earliest" could resolve to the start of the availability window instead. 
This is also a bit out of scope for this proposal, but should be considered for the future.
JSON, and thus JSON-RPC, is a super inefficient data transfer method. For high-throughput chains like Monad, this is becoming a bottleneck in the sense that your cheap ass VPS will just spend its 1 CPU core parsing JSON, making building low-end applications difficult. Same goes for a pure client-side, web browser, application that tries to query Monad.
It might be better to just to have “alternative” data delivery method field in the JSON-RPC request, defaulting to json but then you could ask for protobuf or whatever in the future to get the data out faster. As a bonus this would tax the RPC node provider less.
Common choices inculde Arrow, Parquet, Flatbuffer, Profobuf, various binary JSONs, etc.
Cosmos eco has been using Protobuf since 2017 and it was a good choice for them.
Here is tidbits
Atlassian switched from JSON to Protobuf for one of their services, resulting in: - 75% less memcached CPU usage - 80% smaller data - 33x faster deserialization - a 55% smaller memcached cluster
Some old blockchain data benchmark: GitHub - steelcake/blockchain-data-bench · GitHub
Makes sense @miohtama
I’m trying to picture the typical user who’d benefit most. It feels like (to me) that the majority of the MIP-16 users are mostly indexers on decent hardware where the win is more about cost and backfill speed.
Could you please describe a case where this was an actual problem?
The whole point of this RPC proposal is that the users should not need ot use a third-party indexer. I want to be able to effectively query blockchains without need to pay any centralised third party with massive hardware.
Because otherwise achieving true self-custody, decentralisation, censorship resistance and other high-value blockchain value propositions will be difficult.
I, for example, do not want my wallet to connect to third party indexer just to see what assets it holds because this is a massive violation of privacy.
Here:
Thanks for sharing the manifesto @miohtama, that reads awesome
If im getting this right, MIP-16 moves raw-data lookups such as txs, logs, transfers filtered by address into a standard RPC interface. Curious how you imagine the privacy side though, if a wallet queries a hosted RPC, the provider still sees those queries. Is the idea that self-hosting becomes more realistic
Great question re: "earliest", here are my opinions.
I think the "earliest" block tag should always resolve to the genesis block, even if that block falls outside of the server’s availability range (leading to an error). AFAIK, the "earliest" block tag always refers to the genesis block across all EVM JSON-RPC methods and implementations. I don’t think it would be a good idea to change the meaning here.
If we feel strongly about enabling clients to fetch the server’s availability range, perhaps we should introduce a different block tag, or a different method entirely. However, I’d lean against doing that. A big part of the value of these methods comes from being able to query the entire chain history. While we should specify the behavior of servers that don’t backfill the entire chain, we shouldn’t encourage it.
Thanks for the answer, that makes sense to me!