Key takeaways
- RIKEN has stopped Fugaku, Japan's flagship supercomputer and once the world's fastest, after intruders got into its operations network. The way in was a web page showing Fugaku's operating status, published on the internet.
- Between November 10 and 13, the intruders operated several internal servers and planted malicious programs. They took, or may have taken, the names, email addresses and IDs of up to 4,597 users of Fugaku and of RIKEN's new AI supercomputer, Rikyu.
- More serious is what they found in users' home directories: SSH private keys, access keys and other credentials that researchers had stored there to connect to supercomputers and services outside RIKEN. RIKEN says the intruders have already tried to use some of them against other systems.
- RIKEN acted fast and published a detailed timeline: Fugaku was shut down within about 12 hours of the intrusion being confirmed, and every user's login key was invalidated. But the keys taken from Fugaku have to be revoked at each place they open, by the researchers who put them there.
What happened#
According to RIKEN's notice of November 16:
| Date and time (JST) | Event |
|---|---|
| Nov 10–13 | Intruders use the internet-facing status dashboard as a foothold into Fugaku's operations network, operate several servers and plant programs |
| Nov 12 | Suspicious traffic is detected. Maintenance of the system is outsourced, and RIKEN starts investigating with the contractor |
| Nov 13, 16:33 | Unauthorized access confirmed |
| Nov 13, 20:59 | Reported to the education and science ministry (MEXT) |
| Nov 13, 22:07 | The system used for the intrusion is isolated |
| Nov 14, 04:30 | Fugaku is shut down and outside access cut off |
| Nov 14, 20:20 | Administrative keys replaced |
| Nov 15, 10:51 | Reported to the bodies running HPCI, the network that links Japan's academic supercomputers |
| Nov 15, 16:18 | All running and queued jobs stopped; every user's login public key invalidated; users told to revoke the credentials they kept on Fugaku |
| Nov 16, 17:55 | Reported to the privacy regulator |
RIKEN has not named the vulnerability, the dashboard software or the attackers, and has not given a date to restart.
What was taken#
| Data | Who | Status |
|---|---|---|
| Name, email address, user ID | Up to 4,222 Fugaku users | Taken or possibly taken |
| Name, email address | Up to 375 users of Rikyu, an AI-for-science supercomputer RIKEN is building | Taken or possibly taken |
| SSH private keys, access keys and other credentials users stored for systems outside RIKEN | Not stated | Taken or possibly taken; some already used in attempts |
| Users' research files | Not stated | Taken or possibly taken; no evidence so far that research results leaked |
RIKEN is also checking whether the files contain personal data of third parties, such as research subjects.
Why the keys matter#
Researchers on large supercomputers often work across several of them: they run simulations on one, move data to another and analyze it on a third. To do that from the command line, many keep SSH private keys for the other systems, and access keys for cloud storage and services, in their home directory. It is convenient, and on a well-managed system, the home directory feels like a safe place.
That convenience turns one breach into many. An attacker who can read home directories on one supercomputer collects keys to others, including systems at universities and research institutes in Japan and abroad that were never attacked directly. RIKEN says the intruders have already tried to connect to other systems with some of the credentials. Until every one of those keys is revoked at its destination, the breach is not contained, and only the researchers know where their keys lead.
Supercomputers have been targeted before. In May 2020, several centres in the UK, Germany, Spain and Switzerland, including the University of Edinburgh's ARCHER, shut down systems after intrusions used to install cryptocurrency miners.
What we don't know yet#
- What the dashboard was, who built and ran it, and which vulnerability was exploited
- Whether the contractor that maintains the operations system was involved in running the dashboard
- How many users had external keys on Fugaku, and which systems those keys open
- Whether any of the attempted connections succeeded
- Who the intruders were, and what they were after
- When Fugaku will restart
Our view#
A status page is part of the attack surface. Dashboards that show whether a system is up are built to be public and simple, and are often treated as a convenience rather than a production service. Here, one was the way into the operations network of a national supercomputer. In our view, every internet-facing page should be in the inventory with its owner, its software and its patch status, whatever it is for, and a page that only displays status should not sit on a network that can reach the systems it reports on. It is the same lesson as PHP2Shell: what you do not count, you do not patch.
Keys stored for convenience are the research world's credential sprawl. Researchers did nothing unusual by keeping keys to other systems on Fugaku; it is how many multi-site workflows run. But it means a breach at one centre is a breach of every centre its users reach. Centres can reduce that: short-lived SSH certificates instead of permanent keys, keys protected by passphrases or hardware tokens, and clear guidance that private keys for other systems do not belong on shared machines. We have seen the same pattern in companies this month, where credentials kept in an internal business tool opened the freight forwarding platform Shippio and an analysis tool opened ApplyNow's databases.
RIKEN's notice is the kind we want to see. It gives times to the minute, says what was taken and what may have been, admits that the intruders are already trying the stolen keys, and tells users exactly what to do. It shut the system down within about twelve hours of confirming the intrusion. What it still owes, in the checklist sense, is the way in: the dashboard and the vulnerability, so that other centres can check their own.
What this means for practitioners#
- If you used Fugaku or Rikyu: revoke every key and credential you stored there, at every system and service it opens, and issue new ones. Check those systems' logs for logins from unfamiliar places since November 10. Treat emails claiming to come from RIKEN or Fugaku with suspicion.
- If you run an HPC centre or any system Fugaku users connect to: look for logins using keys registered by Fugaku users, and consider asking them to rotate keys now rather than waiting.
- If you run internet-facing dashboards or status pages: list them, patch them, and make sure they cannot reach the systems they describe.
- For everyone with SSH keys on shared systems: do not leave unprotected private keys for other systems in home directories; use passphrases, agents or short-lived certificates.
Japanese terms at a glance#
| Japanese | Reading | Meaning |
|---|---|---|
| 富岳 | Fugaku | Fugaku (Mount Fuji), RIKEN's supercomputer |
| 理究 | Rikyū | RIKEN's AI-for-science supercomputer |
| 運用系システム | un'yō-kei shisutemu | Operations system |
| 稼働状況 | kadō jōkyō | Operating status |
| ホームディレクトリ | hōmu direkutori | Home directory |
| SSH秘密鍵 | SSH himitsu-kagi | SSH private key |
| 保守管理の委託 | hoshu kanri no itaku | Outsourced maintenance |
| 計算ジョブ | keisan jobu | Compute job |