A gadget is a small, specialized, and efficient “proof component” that can be composed and re-used as part of a larger proof. This is inspired by the definition of gadgets by Campanelli et al..
In our case, a gadget consists of:
Below, we will illustrate this using two examples: calculating the median and the standard deviation.
As a first example, imagine that we want to calculate a median in a ZKP. The prover wants to keep the full input list secret, and only wants to expose the median; the verifier on the other hand needs to be sure the median was calculated correctly. Hence, such a program will typically commit to the input data using a hash and then proceed to calculate the median.
Sorting in the ZKP is prohibitively expensive, as it requires a comparison network. Instead, we developed a gadget that pre-sorts the median outside the proof and just checks whether it was sorted correctly in the proof. Thus, the code in the proof looks like this:
def median<N>(u64[N] vals, u32 n) -> u64 {
for u32 i in 1..N {
assert(vals[i - 1] <= vals[i]);
}
u32 i = n / 2;
return (n % 2 == 0) ? (vals[i - 1] + vals[i]) / 2 : vals[i];
}
This function checks that the input is sorted and then returns the middle result. Note that it is also necessary for the verifier to check whether all values were included (e.g. using message IDs).
We also use the technique of pre-sorting the list to implement functions that take the minimum, maximum, percentiles, etc. We also have a function that returns all distinct elements in a list that relies on the same technique.
Another useful function is the standard deviation. This requires calculating the variance and taking the square root. However, calculating a square root in a ZKP is expensive. It is much cheaper to calculate it outside the proof and verify its correctness in the proof, which only requires a multiplication.
The function to calculate the standard deviation then looks like this:
def stddev<N>(u64[N] vals, u64 stddev) -> u64 {
u64 var = variance(vals);
assert(stddev * stddev <= var);
assert((stddev + 1) * (stddev + 1) > var);
return stddev;
}
The prover must pass in the values as well as the standard deviation that was pre-computed. The function then calculates the variance and checks that the pre-computed standard deviation is correct. The standard deviation may be passed in as a private input.
Note that this requires the prover to calculate the variance twice: once outside the proof to then take the square root to get the standard deviation, and again in the proof to prove that the root of the correct value was calculated. Even though the variance is calculated twice, this is still more efficient than calculating a square root in the proof.
We have developed a library of such gadgets, summarized in the following table. We support all aggregation functions supported by Apache Flink (used by AWS), except those that work on strings or JSON, as well as a few additional functions (median, top/bottom N, any/every).
| Operation | Type signature | Azure | Flink | Complexity | |
|---|---|---|---|---|---|
| Count | []u64 -> u32 |
✅ | ✅ | ✅ | $O(1)$ |
| Count distinct | []u64 -> u32 |
❌ | ✅ | ✅ | $O(n)$ |
| Collect distinct | []u64 -> []u64 |
❌ | ❌ | ✅ | $O(n)$ |
| Sum | []u64 -> u64 |
✅ | ✅ | ✅ | $O(n)$ |
| Max, min | []u64 -> u64 |
✅ | ✅ | ✅ | $O(n)$ |
| Average | []u64 -> u64 |
✅ | ✅ | ✅ | $O(n)$ |
| Variance (population, sample) | []u64 -> u64 |
❌ | ✅ | ✅ | $O(n)$ |
| Std dev (population, sample) | []u64 -> u64 |
❌ | ✅ | ✅ | $O(n)$ |
| Median | []u64 -> u64 |
❌ | ❌ | ❌ | $O(n)$ |
| Top N, bottom N | []u64 -> []u64 |
❌ | ✅ | ❌ | $O(n)$ |
| Top distinct N, bottom distinct N | []u64 -> []u64 |
❌ | ❌ | ❌ | $O(n)$ |
| Rank | []u64 -> u32 |
❌ | ❌ | ✅ | $O(n)$ |
| Dense rank | []u64 -> u32 |
❌ | ❌ | ✅ | $O(n)$ |
| Percent rank | []u64 -> u32 |
❌ | ❌ | ✅ | $O(n)$ |
| Cume dist | []u64 -> u32 |
❌ | ❌ | ✅ | $O(n)$ |
| Row number | []u64 -> u32 |
❌ | ❌ | ✅ | $O(n)$ |
| Ntile | []u64 -> u64 |
❌ | ❌ | ✅ | $O(n)$ |
| Percentile | []u64 -> u64 |
❌ | ✅ | ✅ | $O(n)$ |
| First, last | []u64 -> u64 |
❌ | ✅ | ✅ | $O(1)$ |
| Lead, lag | []u64 -> u64 |
❌ | ❌/✅ | ✅ | $O(n)$ |
| Any, every | []bool -> bool |
❌ | ❌ | ❌ | $O(n)$ |
| Bitwise AND, OR, XOR | []bitmap -> bitmap |
❌ | ✅ | ❌ | $O(n)$ |
| ❌ |
([]string, string) -> string |
❌ | ❌ | ✅ | |
| ❌ |
[](string, T) -> JSON |
❌ | ❌ | ✅ | |
| ❌ |
[]T -> JSON |
❌ | ❌ | ✅ |
Our library relies on a few tricks:
There are a few limitations to our approach:
Furthermore, in the future we could add some additional gadgets for common streaming operations. In particular parsing JSON (or Protobuf or other data formats) would be useful. We could build a small compiler that, from a schema of the message, automatically builds a proof that checks that the message is well-formed and extracts the relevant fields.
In the end, using these gadgets a developer can build an application in ZK, by combining the gadgets that are implemented in an efficient and secure way, and without needing to re-invent the wheel every time.
The gadgets are open sourced on GitHub.
In the longer term, we could also create a compiler that takes a SQL query (or a dataflow script) and compiles it to a ZKP. This would allow developers to write their queries in a simple, high-level language and then have them executed in a ZKP, without needing to know the intricacies of ZKPs but still benefiting from their confidentiality and integrity properties.
Firstly, let’s briefly repeat what Zero-Knowledge Proofs are. A Zero-Knowledge Proof (ZKP) is a cryptographic technique that allows:
For a more detailed description of ZKPs, we refer to this blog post by Vitalik Buterin explaining the maths behind zk-SNARKs, this more high-level article explaining what ZKPs are, and this YouTube video explaining ZKPs at different levels of abstraction.
In the rest of this post, we will be using the ZoKrates programming language to create ZKPs. This example program proves knowledge of the factors of a (large) number:
def main(private field a, private field b) -> field {
assert(a > 1 && b > 1);
return a * b;
}
This program takes in two inputs a and b, which are numbers in a finite field (up to ~255-bit). The keyword private indicates that these inputs are kept confidential. The program first checks that both a and b are greater than 1 and then returns their product.
After this program is executed, a proof is generated that contains the result of the computation, but not the (private) inputs. This proof can be verified to check that the computation was executed correctly, without revealing the inputs.
When executing the program with the inputs
123456789and987654321, the proof ends up looking like this:{ "scheme": "g16", "curve": "bn128", "proof": { "a": [ "0x2438520bf50f1ef4f0cbd71ad04069c72e4683b88f5f2becb98bbf7c0c101582", "0x1940ec3fb0df60726e6bc95e196171e826b00453a106a6085d61d4327ea445cb" ], "b": [ [ "0x2403a661a61aa80b86fe5515547e99e9d8139d5f892c9a29b03f0e2eda82677b", "0x0cc2321418da1284478f2a9397d279eddecfb631965981c934d065f6c058e2f1" ], [ "0x2865222599023b1bb0776c8c5eb23b780d0cf2659d4ccb4e511e2df22b9612f2", "0x195051b7b0cb3aa2577aa3b99376bd44b8424125db2350015331288cdd2be045" ] ], "c": [ "0x05759e3efb49a3dc8c11cf7fdb4cbd152dd64b1d233f8a9da03a03afac532a4d", "0x286af8c6919b3053acef0fad338b38bc91e92071d6fe3757cc083d79639e7cf6" ] }, "inputs": [ "0x00000000000000000000000000000000000000000000000001b13114fbff5385" ] }Here, the
inputscontains the public inputs and outputs, in this case only the product of the two secret factors (121932631112635269 in hex). Together with the other numbers and a verification key, this forms a proof that can be verified by anyone. The proof is tied to this particular program.
Similarly, the following ZoKrates program proves that the prover knows the pre-image of a SHA256 hash, without revealing the input (relying on the sha256 function of ZoKrates’ standard library):
def main(private u8[256] input) -> u32[8] {
return sha256(input);
}
In other words, a proof generated by this program shows that the prover knows an input that hashes to a particular hash. This can be used to show that a prover knows a password, without revealing the password.
Hashes are also used to create a commitment to secret data: multiple programs can prove that they are using the same input data, while keeping this data secret, by each calculating and returning the hash. If two programs output the same hash, we know they must have used the same input to the hash, but we cannot deduce this (secret) input because the hash is a one-way function.
The program that calculates the SHA256 hash of a 256-byte input is quite slow. (The program corresponds to a circuit of 145560 constraints.) On a MacBook Pro with a 2.3 GHz 8-Core Intel Core i9 from 2019:
Moreover, the performance characteristics of ZKPs are non-obvious. As these programs are compiled to arithmetic circuits that work on a finite field, their performance is quite different from ‘normal’ programs. For example:
if, both branches are executed. This also means no short-circuiting of && and ||.u8, …, u32, u64 may need to be emulated using operations on their individual bits.When programming ZKPs, it’s very easy to make mistakes, which can have severe consequences.
For example, when writing a program to prove knowledge of the factors of a number n, a first attempt might be:
def main(private field a, private field b) -> field {
return a * b;
}
However, this program allows the prover to generate a trivial proof by picking a = 1 and b = n, hence allowing the prover to generate a valid proof without knowing the factors of n. This breaks the computational integrity we expected as verifier.
As another example, the program above that proves knowledge of the pre-image of a SHA256 hash actually contains a mistake (depending on the use case): as the input is not salted, the verifier may be able to guess the input by brute-forcing all possible inputs. This breaks the confidentiality that we expected as prover. A better implementation would have been:
def main(private u8[256] input, private u8[256] salt) -> u32[8] {
return sha256([...input, ...salt]);
}
Such bugs have been observed in practice [1], [2], appearing in popular projects like the Bulletproofs paper, zcash, and Circom’s standard library. These bugs are hard to find and hard to fix, as they require a deep understanding of the program and the cryptographic properties of the ZKP. Note also that many existing bug detection and static analysis tools focus on finding the first kind of problem (bugs that produce incorrect outputs), and not accidental leaks of confidential information.
Finally, the ZoKrates language (as well as other languages for ZKPs) is quite limited. For instance, there is no support for floating-point numbers or strings. The standard library is also very limited, with support for (1) hashing, (2) operations on elliptic curves, (3) EdDSA signature verification, and (4) some functions for type conversion; but not much else.
In this post, we described the main challenges of developing ZKPs: their slow and non-obvious performance, and how easy it is to introduce bugs that break confidentiality or computational integrity.
In a next post, we will introduce our current solutions for these problems, as well as our vision on how to improve this more generally in the future.