Zero-knowledge Machine Learning (ZKML)

Zero-Knowledge Proofs

On the algorithm side, Zero-Knowledge Proofs (ZKPs) are crucial in achieving the above objectives. They provide the means to confirm the authenticity and integrity of the models run by nodes, thereby providing trust among users and stakeholders in the decentralized system. Specifically, a ZKP allows a prover $\mathcal{P}$ to convince a verifier $\mathcal{V}$ that a statement is true without revealing any information beyond the statement's validity. For model verification, $\mathcal{P}$ demonstrates that $\mathcal{M}$ executed correctly on input $x$ to produce output $y$, without revealing $x$, $y$, or $\theta$.

Implementation: Zero-Knowledge Machine Learning for Private Models

Nesa supports Zero-Knowledge Machine Learning (ZKML) for selected private models such as Convolutional Neural Networks (CNNs), and we look forward to working on customized solutions for our clients. Applying ZK ML to smaller, private models offers a secure way to maintain the confidentiality of sensitive data and model parameters across decentralized systems.

To apply ZKPs to smaller, private models such as on neural networks, linear models, or decision trees, we can outline the approach as follows:

• Model Execution: Consider a simple linear model for demonstration:

  $$y = \mathbf{w} \cdot \mathbf{x} + b$$

  where $\mathbf{w}$ represents the weight vector, $\mathbf{x}$ is the input vector, and (b) is the bias.

• Proof Construction: The prover $P$ constructs a proof that they have calculated $y$ correctly using the model parameters $\mathbf{w}$ and $b$, without revealing these parameters or the input $\mathbf{x}$. The proof must demonstrate that:

  $$y = \mathbf{w} \cdot \mathbf{x}+ b$$

  is computed correctly, aligning with the agreed-upon model structure and parameters.

• Verification Process: The verifier $V$, upon receiving the proof, checks its validity through a verification algorithm $\mathcal{V}$ that confirms whether the provided (y) is consistent with the execution of $\mathcal{M}$($\mathbf{x}$; $\mathbf{w}$, $b$) without learning any specifics about $\mathbf{w}$, $\mathbf{x}$, or $b$.

• Zero-Knowledge Property: The crucial aspect of ZKPs in this context is ensuring that no knowledge other than the correctness of the model execution is leaked. This is typically achieved by structuring the proof to obfuscate the details using cryptographic techniques such as commitment schemes or garbled circuits.

By leveraging ZKPs, organizations can validate the integrity of computations in sensitive applications without exposing the underlying data or model specifics, enhancing trust and security within decentralized systems. Nesa is committed to expanding its ZK ML capabilities to include more model types and customized solutions tailored to our clients' specific needs.

Visioning: Zero-Knowledge Machine Learning for Large, Public Models

First, we argue that since open foundation models' parameters are already accessible from all the parties, the verification can go in a much more efficient way other than ZKML, in which costs increases significantly with model complexity.

Specifically, in the decentralized setting, ZKPs face challenges for large language models (LLMs). The primary challenge stems from the computational complexity and scalability of ZKPs when applied to the vast parameter spaces typical of LLMs, represented by $\mathcal{M}(\mathbf{x}; \theta) = \mathbf{y}$. Generating and verifying ZKPs for models with billions of parameters can be prohibitively expensive regarding computational resources and time, making real-time verification impractical. Moreover, the dynamic nature of LLMs, which may require continuous updates and retraining, further complicates the generation of ZKPs.

Additionally, the complexity of the inference tasks performed by LLMs introduces additional challenges in formulating ZKPs. The deep layering and non-linear operations in LLMs, e.g., transformers, represented mathematically as a composition of functions $\mathcal{M} = f_n \circ \cdots \circ f_1$, require sophisticated ZKP schemes that can efficiently handle such complexities. Existing ZKP protocols may not readily adapt to the specific requirements of verifying LLM inference while preserving privacy guarantees.

ZKML in the Nesa System

In summary, ZKPs present an ideal and visionary approach for ensuring model integrity and establishing trust within decentralized inference systems utilizing LLMs. The theoretical appeal of ZKPs lies in their potential to verify computations without revealing the underlying data or the specific operations performed, aligning perfectly with the privacy and security requisites of LLM applications. Thus, we support ZKML for private models with the highest security ensurance.

For large public models, the realization of ZKML faces substantial challenges today, primarily due to the computational complexity, scalability issues, and the sophisticated nature of LLMs. As such, while ZKPs remain a compelling direction for future research and development, enabling their use with LLMs and other large models in decentralized systems is a goal we are actively researching and developing.