HPC Info - Latest Comments

Re: Amazon’s HPC cloud offering: “Cluster Compute Quadruple Extra Large” instances

jllomster — Thu, 17 Mar 2011 11:46:32 -0000

Blows my mind. I'd love to hear more about this and also your thoughts on the movement towards neural networking.

Re: DARPA Challenge: billion-way parallelism, 1 PFLOPS system in one rack, 57 kW max power

Manyone64 — Fri, 14 Jan 2011 08:28:52 -0000

I appologise for my delay in commenting on this earlier, although I was discussing a solution to this exact problem with IBM in early July 2009, which ended in a suggestion to go back to Cray as I had in the past. Which I didn't, due to my understanding that I don't fit the profile of someone who would normally solve such a problem and therefore not given any support.

I think the main problem people have with my design is due to lack of recognizable parts, all of the primary components are sourced through OEM suppliers using standard form factors, the only difference is that I designed my own housing. Instead of conforming to the outdated rack mounted system. This is not something IBM or the like would be interested in, due to thier over investment in rapidly redundant technology. I guess they now know what the rest of us have felt. You spend all your money on the latest tech, only to find it made redundant the following week. The solution to this problem requires complete re-tooling, something I am very familiar with and comfortable in doing.

Currently my design would have the following basic specs, which would require a new OS written from the ground-up in x86 machine code. But I believe this cannot happen until the hardware has been built, then the code can be written and optimized to extract every last flop.

Overall System Specs:
64 Nodes(solidstate components only, no moving parts).
64 port FSO sub-meter interconnect.
1 x 64 Channel 32KW Power supply(Using DMX512 Protocol)
Intergrated Conductive Cooling solution(Using 36KW Water Chiller unit)
Dimensions: Similar to a 44 Gallon drum.
Approx Weight: 250Kg

Per Node specs:
2 x Nvidia GTX280
1 x 2.5Gb/s Optical I/O
1 x i5-750
1 x Power PCB(DMX512 interface)
1 x 4 Slot PCIe (x16) Backplane PCB

Retail Cost:$250,000AUD (budget set before design started, to be more affordable for smaller businesses.)

Estimated Performance: 120TFlops

Cost per GFlop: $2.09

GFlops Per Watt: 3.29

Watts per GFlop: .3

This design needs to be built, so that it can evolve and become something better. I know this design isn't perfect, but it's a start in the right direction. Given real support, this design could be built and proven in 9 - 12 Months, at a fraction of the cost of the Roadrunner. Beep beep!

But I am not even sure this comment will be read, at least not by anyone who could change my situation!

Re: Benchmarking the Cloud for Genomics

Gary Stiehr — Sun, 22 Nov 2009 21:05:30 -0000

Thanks for the comment, Nick. They discuss a scenario of transferring the input data with parallel EC2 nodes into S3 directly from EBI (see p.7), which I'd think has a relatively high bandwidth connection to the Internet. The transfer (of the compressed dataset) took 1h15m at about 187 Mb/s and of course you incur EC2 charges for decompressing that data. It wasn't mentioned about how long that decompression took.

Re: Benchmarking the Cloud for Genomics

nickloman — Sun, 22 Nov 2009 08:12:19 -0000

Interesting Gary. I see the major challenge being getting your data in the cloud in the first place. The datasets are huge and Internet connections still relatively slow.

Re: Scalable Staging of Large Datasets to Many Compute Nodes

Gary Stiehr — Wed, 24 Jun 2009 23:30:44 -0000

In terms of using bittorent, we've done some initial tests with good initial results using rtorrent, which uses a tracker for directing download activity. I think the trackerless variety of bittorrent clients are harder on the network. Although I am not an expert with bittorrent.

How could Xcp help in alleviating the load on the original file server when distributing a large data file to many nodes?

Re: Scalable Staging of Large Datasets to Many Compute Nodes

Joe Landman — Fri, 19 Jun 2009 21:07:05 -0000

A few years ago, I had written a code called mcp to do broadcast copies. It was an O(1) copy based upon network multicast copy. I later started xcp which did something similar, without broadcast/multicast.

This predated bittorrent by a bit. I had read some reports of people looking at bittorrent for large data motion. It wasn't very efficient at moving large data files in a cluster as I remember. Some of our customers have had significant problems with the Rocks installer. It turns out that it interacts badly with a number of switches, or switch topologies for larger clusters.

Xcp has a number of neat features, but we never finished it (in the sense of had it in a done state ready for people to play with and use). Xcp was designed to reliably transfer data on possibly unreliable networks.

Bug me about this some time, and if you are interested, we might work on getting a version going you can use.

Re: Using HPC to Understand Swine Flu

Chris Clark — Tue, 09 Jun 2009 12:11:04 -0000

Gary,
I thought this was a great post and look forward to the potential here.

Clark

Re: Sequoia: 20 Petaflops, 1.6 million cores, 1.6 Petabytes RAM, 6 Megawatts

Brock Lesnar — Sun, 08 Feb 2009 00:19:44 -0000

You *will* be paying the electrical bill on it... So... Get used to it. :)

Re: Sequoia: 20 Petaflops, 1.6 million cores, 1.6 Petabytes RAM, 6 Megawatts

Jeff Manes — Sat, 07 Feb 2009 15:54:58 -0000

Wow, I'd hate to pay the electric bill on bad boy. Maybe the contractor would throw in a windmill or a solar panel in the deal.

Re: Sequoia: 20 Petaflops, 1.6 million cores, 1.6 Petabytes RAM, 6 Megawatts

Greg Gurevich — Fri, 06 Feb 2009 15:41:44 -0000

Just yesterday it seams we crossed 1 Petaflop.

Greg Gurevich

Re: What do you consider a “large file system” and are you worried about fsck times?

Gary Stiehr — Fri, 23 Jan 2009 23:51:41 -0000

See http://www.linkedin.com/ans... for some interesting replies to this question.

Re: TACC's Ranger Cluster - Updated

Gimlet — Tue, 31 Jul 2007 16:06:21 -0000

Sun CEO Jonathan Schwarz recently blogged on the specs of the TACC Ranger. Not surprisingly, Sun's new HPC offerings featured prominently in his post.

Gary, I'm sure you could put one (or more!) of those Magnums to good use!

Off-topic for this blog, but does anyone else think that the Pegasus blades are also meant to be sold to the likes of Google?

Re: TACC's Ranger Cluster

Gary Stiehr — Tue, 03 Jul 2007 00:33:58 -0000

Hi Jay, thanks for the comments. I look forward to seeing the updated page.

Re: TACC's Ranger Cluster

Jay Boisseau — Mon, 02 Jul 2007 17:21:58 -0000

I'm the director of TACC and the PI of the proposal. I apologize for the inconsistent information. Our plans changed (improved!) mid-stream but we could not update them publicly until we receive formal approval of the changes, and by then we, um... forgot to update that page. I will see that the Ranger on our web site gets updated right away with the answers to all of the technical questions above.

Re: TACC's Ranger Cluster

Gary Stiehr — Sat, 30 Jun 2007 01:16:22 -0000

Thanks for the info. It is interesting how much conflicting information there is throughout the various articles. Perhaps some of them were written before the final configuration was decided upon. Although I'd expect TACC's description (13,152 processors) and Sun's description ("over 15,000" processors) to be the closer. Well, what's a couple thousand processors here and there?

I'll have to watch for info on how they've configured the x4500s.

Re: TACC's Ranger Cluster

gimlet — Sat, 30 Jun 2007 00:37:17 -0000

Sun has a blurb on their website about the TACC Ranger

The Ranger Cluster and TACC

The first petascale implementation of the Sun Constellation System is the Ranger HPC cluster, which was developed jointly with the Texas Advanced Computing Center (TACC) of The University of Texas at Austin. When it is fully deployed on the TeraGrid national network of supercomputers in late 2007, it is expected to be one of the most powerful general purpose computing platforms in the world at over one-half petaFLOP peak performance.

The Ranger cluster will deliver 1.7 petabytes of storage using the Sun Fire X4500 data servers, the highest density available. Once completed, the TACC installation will consist of over 80 Sun Constellation System racks of computing power totaling over 15,000 quad-core microprocessors, all connected by Sun's new high density, 3456-port InfiniBand switch. Sun Grid Engine will be used as a resource manager to dynamically allocate compute resources to applications.

Re: Petascale Storage / Storage-over-IP

Gimlet — Wed, 28 Jun 2006 16:21:22 -0000

I posted too soon. Sun has released the 6/06 edition of Solaris 10, which includes ZFS.

Re: Petascale Storage / Storage-over-IP

Gimlet — Sun, 25 Jun 2006 23:50:25 -0000

ZFS has been released as part of Solaris Express, which is based on the OpenSolaris project.

There seems to be at least one grid using it, operated by Joyent. Their grid is for pushing web apps, not scientific calculations, but storing and moving bits (or terabytes) around is pretty much the same everywhere. They have a useful slide presentation online.

Re: Petascale Storage / Storage-over-IP

Gimlet — Tue, 23 May 2006 23:53:31 -0000

Welcome to blogging...