The Azure company said that attackers used machine-learning clusters rented by customers for cryptocurrency mining at the customers’ expense
Machine-learning tasks involve a tremendous amount of computing resources. The attackers took advantage of this fact and generated large amounts of the currency while the customers made use of the clusters. The misconfigured node made the attack easy for the attackers.
Microsoft said, “the infected clusters were running Kubeflow, an open-source framework for machine-learning applications in Kubernetes, which is itself an open-source platform for deploying scalable applications across large numbers of computers. Compromised clusters were numbered in the “tens”. Many of them ran an image available from a public repository, ostensibly to save users the hassle of creating one themselves. Upon further inspection, Microsoft investigators discovered it contained code that surreptitiously mined the Monero Cryptocurrency.”
Execution
Once investigators discovered the infected clusters, the next step was figuring out how the machines were compromised.
The set-up of the system ensures that access to the administrator’s dashboard and control of Kubeflow is via istio ingress. Istio ingress is a gateway at the edge of the cluster network. It ensures that no unauthorized changes takes place in the cluster.
Gaining access to the dashboard is just the first step. After this, the attackers explore several options for deploying a backdoor in the clusters.
One of such options is the placing of a malicious image inside a Jupyter Notebook server.
A Security-research software engineer in the Azure Security Center, Yossi Weizman; said that the users unknowingly change a setting, which invariably gives attackers access. In the post released on Wednesday, he wrote “we believe that some users chose to do it for convenience, without this action, accessing the dashboard requires tunneling through the Kubernetes API server and isn’t direct. By exposing the Service to the Internet, users can access the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, allowing anyone to perform operations in Kubeflow, including deploying new containers in the cluster. Azure Security Center has detected multiple campaigns against Kubernetes clusters in the past that have a similar access vector; an exposed service to the Internet. However, this is the first time we have identified an attack that specifically targets Kubeflow environments specifically.”
The company’s post gave users multiple techniques for checking if the clusters are vulnerable.
