Decentralized Learning in Untrusted Environments

Description

Federated learning has emerged as a privacy-preserving method for training a machine

learning model. Training a model in federated learning is performed by clients that

have local data, and a server manages the training process. The local data may

contain sensitive information that should not be shared to third parties. In this regard,

federated learning has been considered a secure training method that protects clients’

data. However, in recent studies it was shown that the clients’ data can be reconstructed

from their gradient updates sent to the server for aggregation.

We present a secure and scalable protocol for aggregation, random gradient mixing.

By exchanging gradient updates between clients before they are sent to the server, the

protocol enables anonymization and obfuscation of reconstructed data. The random

gradient mixing protocol provides security not only against a malicious server, but also

against honest-but-curious clients.

The security level of random gradient mixing depends on the hyperparameters. We

present the experimental result and evaluate the quality of reconstructions and the

performance of the models with different hyperparameters for random gradient mixing.

The choice of hyperparameters also affects the robustness of the random gradient

mixing protocol against clients dropping out. With a small range where a scale vector

is sampled from, it can be perfectly robust to dropouts. In order to mitigate potential

training failure due to dropouts, we also discuss an additional recovery phase.

The protocol is also efficient in communication. Since the communication cost

increases linearly with the number of clients, the protocol is expected to have a strong

scalability in practice.