Motherboard

To most humans, the PlayStation3 is for nerding out with shooter games like Quantum Theory or Dead Space over frozen Milky Way bars. To astrophysicist Gaurav Khanna those words aren’t games but the stuff Nobel prizes are made of. When Khanna fires up his PS3 — or all 16 of them — he’s first-person shooting for the stars.

In 2007, the researcher from the University of Massachusetts Dartmouth (not the Bollywood hunk) decided he could no longer afford the $5000 in quarters needed to use an institutional supercomputer to study gravitational waves emitted by super-massive black holes. Like a thrifty gamer trading the arcade for a home-built computer console, he came up with his own solution: build a supercomputing cluster out of PlayStation3s, which contain a little-known but mighty processor called the Cell and can be outfitted with Linux. Dubbed the PS3 Gravity Grid, Khanna’s supercomputer made of video game consoles has helped him advance his celestial research at Sony’s cutrate prices, made him a pioneer in the nascent PS3 supercomputing field, given him new excuses to play video games, and earned him phone calls from the Air Force, which recently put 2,200 of the consoles on its holiday wish list for its own PS3 supercomputer. We spoke to him by email about how he went from Sony to so neat, the unending wonders of open source, and the unknown wonders of black hole physics.

Motherboard: What is a supercomputer? And how does a system like yours achieve that exalted title?
Gaurav Khanna: Thats a good question. Its difficult to have a very tight definition of a supercomputer, because of the rapid advances in the performance of modern computers. Essentially, what was a cutting-edge supercomputer yesterday is sitting in everyone’s lap today.

My viewpoint is that a supercomputer is a machine that has compute capabilities that are far beyond those of a typical desktop or workstation at that time. From that stance, stringing along a number of workstations and running them in parallel to obtain a large performance gain in an application, would certainly qualify as a supercomputer. In that sense, if a gaming system is able to deliver performance that is far ahead of the present day desktop, then it can be considered a supercomputer too.

What inspired the use of PS3s specifically to build the gravity cluster?
At the time the PS3 was released, I was working on an extremely computationally intense problem in my research area of black hole astrophysics. I didn’t have enough supercomputing time on national supercomputing facilities to be able to study the problem, so I started to look into “creative” solutions ;-).

There are three main things about the PS3 that make it extremely attractive as a compute node in a large cluster: 1. Open Platform — It can run Linux and therefore function as a traditional workstation (with all the open-source software that is essential for scientific computation accessible!), 2. Cell processor — It is extremely powerful (thanks to the legendary STI Cell Broadband Engine) and 3. Commodity — It is very inexpensive and readily available in large quantities.

Cell processor? STI Cell Broadband Engine? Can you describe what’s significant about that acronym-heavy hardware?
The Cell BE was designed from scratch, with high-performance in mind. Nearly half a billion dollars of R&D was invested into this architecture by STI (Sony-IBM-Toshiba). It has a unique hybrid design that includes elements from traditional CPUs and GPUs. In essence, it has advantages of both processor designs — a fast CPU for serial tasks (PPE) and multiple vector compute engines (SPEs) for efficient computation — all on the same chip, so that these elements can exchange data with each other at extremely high-speed.

How surprised are you at the reaction your cluster has been getting from the public and from the academic community?
Yes, a little bit. We have received a lot of media attention over the past 2 years in the US and abroad. In addition, my work got a lot of attention on my own campus and attracted a number of strong students to my group. And then the publicity also helped build strong connections within Sony, IBM and related industry.

What involvement has Sony had, and what have they said about your research? Are they using this to test for some new astrophysict role-playing-game?
Sony has been extremely kind and supportive of this project from the beginning. They donated a number of units right in the beginning (mid 2007) to help us get started. In addition, they helped by offering engineer consultation time and inside information, whenever possible. I continue to stay closely in touch with the folks in the Sony R&D. They are excellent people that bend over backwards to help! Couldn’t have done this without Sony’s help.

I believe they are happy with what we have been able to achieve with the PS3 hardware. Of course, their focus is games and thus they have limited involvement. But they help when they can.

What was your earliest experience with a supercomputer like, what was it, and how big was it?
I guess that would be in the late-90’s, when I was in graduate school at Penn State. I used an SGI Power Challenge, that had maybe 16 processors, each clocked at about 200 MHz. Things have changed quite a bit since then.

Why the “gravity” cluster? Is that a reference to your research?
Absolutely. I’m a gravitational physicist, and I work on problems within Einstein’s General Relativity Theory. Its this theory that grants us our present understanding of black holes, cosmology and gravitational waves.

My wife actually named the cluster. I wanted her to do so; she is the one who braved those long lines in late 2006 (when the PS3 was released) to get me one for my experimentation!

The thing is so inexpensive I heard, because Sony subsidizes it or sells below cost, making up the loss with sales of accessories and games. So science is benefitting from gamers’ willingness to shell out lots of money for those things?
Exactly right! The PS3 is heavily subsidized, thanks to the marketing model commonly employed in the gaming industry. And indeed, gamers are helping science, not just in the context of the PS3, but also with the advancements in graphics cards, network communication hardware and many other technological areas.

Have you heard of other unintuitive uses of conventional technology? And are there other new or upcoming applications of these kinds of clusters that you know about?
Another one that comes to mind is the current widespread use of Nvidia GPUs for scientific computing. Graphics cards (GPUs) pack a tremendous amount of computing power and that demand has again been driven by the gaming industry. Nvidia realized this potential market for their products a few years ago and invested in CUDA and their Tesla & Fermi product line that is completely focussed on high-performance computing.

Years ago the University of Illinois built a compute cluster using PS2s. I didn’t know about it until relatively recently, actually. I have heard many other ideas that involve clusters of PS3s that are currently in use or are planned. These vary from molecular dynamics simulations to cracking passwords and more. Of course, then there is the Air Force that is building a 2000+ PS3 cluster as well…

Right. What has the Air Force told you about its interest in PS3s? And when they called you up, what kind of advice did you give them?
Alex, I probably shouldn’t get into what AFRL is doing and planning with those PS3s in this interview. I’d suggest we avoid these questions…. I’m not trying to be secretive or anything. I’d rather not talk about what others are planning to do.

Uh huh. What exactly do you think would happen to you if you revealed what the USAF were doing?
I’d be visited by the men-in-black. No, seriously. Its nothing to do with the Air Force. I just don’t like to say something on someone else’s behalf. I may misrepresent them.

Yes — accuracy first! Meanwhile, I know this kind of hardware comes with a strict prohibition from overseas sale, probably because of military concerns. But wouldn’t it be easy for Beijing to buy up as many PS3s as they wanted, without any kind of background checks?
I think anyone can buy a good number of PS3s very inexpensively and put them in their basement and do some potentially damaging thing. I hope that doesn’t happen, of course, but its entirely possible. I’m assuming that our federal government has an eye on such things and will prevent any misuse of such powerful technology.

Speaking of misuse, are you a gamer? What video game systems have you/do you use for fun?
Not really :-(. Since I’ve had the PS3s around, I’ve indulged in a game or two now.

Which games do you like? Are there other gamers in your lab? Can those PS3s still be used for gaming, even in that array?
I enjoy Tekken and almost all car-racing games. And then, there is LBP which is, of course, no less than awesome — perhaps the most creative game I have ever seen. I also thoroughly enjoy Rock Band with my kids. Yes, there are plenty of gamers, especially in the student population in the department! I don’t know very much about networked games, but I’m sure the PS3 cluster could be used that way. Actually, we’ve talked about doing a LAN party with exactly that in mind. It hasn’t happened so far. Maybe we’ll try this year.

Is this the kind of thing that I could build at home? Have you heard of other interesting applications?
Absolutely. You could build a similar cluster of any size. Even 2 PS3s could be networked similarly. Please feel free to visit ps3cluster.org for a DIY guide. There are now several applications using PS3 clusters for gains in performance all over the world. Examples include, protein folding, cryptography, code-cracking, etc.

Where is your own research focused at the moment? Tell me about how the cluster has helped your research?
I work on various computational problems within the area of black hole physics. They all involve modeling the process of gravitational wave emission from black holes; the work is relevant to the National Science Foundation’s LIGO project (ligo.org) and ESA/NASA’s LISA Mission (lisa.nasa.org). Essentially, when a star or small compact object gets swallowed by large black hole (such as the one in the center of our galaxy), a gravitational wave is emitted, carrying all sorts of valuable information that is likely to be picked up by the LISA space-antenna. That is one of the astrophysical processes that I study using high-performance computers.

The PS3 cluster that I built over 2 years ago has helped further this research considerably. So far there have been 5 research publications to date that have wholly or partially used data generated by the cluster. And it continues to be an incredible resource and is still in full use even today. Anyone can learn about the cluster and the related research by visiting my website.

What about the information emitted by black holes makes it significant to our understanding of physics? What significant, new qualities you have learned about black holes in the past few years?
Thats a loaded question. Let me answer it this way. Gravity is the oldest known, yet least understood force. It would be fair to say that all the other forces in Nature are reasonably well understood at a very fundamental level. But gravity has never quite fit in the same framework. Therefore, if you want to really understand gravity in depth, you obviously have to study phenomena in the Universe that involve a very strong gravitational influence. That means studying massive and dense objects like black holes, neutron stars, etc. that have very strong gravitational phenomena associated to them. For this reason, and also to understand more about the astrophysical objects that make up our cosmos, its important to study the gravitational waves that are emitted by such systems.

There are many incremental things that have been learned by the efforts made by my community of physicists. Largely, the recent efforts have focused on building detectors that can directly observe these gravitational waves and thus being able to do astronomy with them. These elusive waves haven’t yet been detected, but the hope is that they will be soon. If anyone is interested in learning more about gravitational waves, I suggest they visit the websites of LIGO (ligo.org) and LISA (lisa.nasa.gov).

I’ve been reading a lot about CERN’s recently restarted Large Hadron Collider recently. What do you know about the creation of quantum black holes in the LHC, or is that bunk? What other kinds of findings are you looking forward to most in terms of the LHC experiment?
I don’t believe that black holes will be created at the LHC. Some people make that argument on the basis on some highly speculative ideas; and even if that were true, these microscopic black holes would “evaporate” near instantaneously and would thus never be detected, let alone do any harm!

I think the big thing to watch for at the LHC is the Higgs particle, of course. That discovery would validate an extremely large volume of work over the past several decades. Of course, one can always hope for an unexpected happening that surprises the entire community of physicists!

Here’s something I want to institute on Motherboard: What’s your biggest question?
I think for me, the biggest question is whether science will be ever be able to completely uncover all the laws of Nature. As we gain more understanding, will that open up questions or will Nature completely reveal itself at some point?

Lastly, as a holiday gift, will you take any old PS3 at your lab? Even if it’s covered in pizza cheese and has a slightly broken disc drive?
Absolutely. We’d be happy to accept even malfunctioning PS3s (for parts). We don’t even need the blue-ray drive to be functioning for a console to be part of our cluster. If anyone would like to consider such a donation, they should directly contact me. We’d appreciate it very much.

Hmm… the afternoon delight of some loafer on Craigslist or humankind’s understanding of black holes. That’s a tough one.

— INTERVIEW BY ALEX PASTERNACK