Cluster building at UWC

students assembling compute cluster

Motse, Eugene (hidden) and Saeed assembling a Dell R710 they’re going to use in their cluster

Last year the Centre for High Performance Computing (CHPC) ran a student cluster building competition for the first time, alongside their national meeting. The winning team progressed to the International Student Cluster Challenge in Leipzig and won top honours there. Observing the teams at work last year convinced me this is something we need to introduce to UWC, so his year when David Macleod from the CHPC’s ACE Lab is announced the second installment of  competition I contacted Computer Science to make sure we had a team. From that side Reg Dodds is facilitating things, and their sysadmin, Daniel Leenderts, is offering a helping hand. The team is being mentored by Motse Lehata, and includes Warren Jacobus, Saeed Natha, Nicole Thomas and Eugene de Beste.

On Tuesday Long and myself wandered over the CS to observe and assist with the unpacking and installation of the practice cluster that Dell had sponsored. I’ll be thin on the technical details in case they don’t want it shared, but it provides enough hardware for installing and testing an operating system and applications for benchmarking. I’m hoping to use this cluster building as an opportunity to get students (and faculty) interested in building cyberinfrastructure as an area for research and maybe even future careers. After all, right now I’ve got the distinct impression that the small number of people I know that run the (mostly Linux) servers that power South African e-Research infrastructure ended up in that career path largely by accident. With big international projects like the SKA and H3Africa coming on stream in the next few years, we’re going to need a much large pool of scientific computing, High Performance Computing, scientific workflows (my personal research bugbear), data curation, storage and re-use and so on expertise. Right now, as far as I can see, there is no decent curriculum out there to train these people, something that I’m trying to address in my small way as part of the H3ABionet, and there is no clear track through the educational institutions into the research infrastructure (as opposed to pure research) side of things. Its gotta change!

e-Research Africa 2013

Quite by accident I ended up attending (and speaking at) the e-Research Africa 2013 conference. This was held in Cape Town, and largely organised, I gather, by Ed Rybicki and Sakkie Janse van Rensburg, from UCT. Ed is the Academic Liason to the UCT Research Portal project, and Sakkie is the Executive Director of ICTS (basically Campus IT services) at UCT. Sakkie was at University of the Free State previously (which in my mind is currently most notable for providing employment to Albert van Eck, one of the more experience HPC admins I know).

The conference started with a keynote from Paul Bonnington, the Director of e-Research at Monash University, and what struck me about Paul’s presentation was the careful attention given to the human and institutional factors that got into e-Research productivity. The topic was “eResearch: Building the Scientific Instruments of the 21st Century – 10 Lessons Learned”, and it set the tone for the conference with a few key message:

  1. e-Research infrastructure is built for an unknown future. Paul gave the example of PlyC lysin, a novel bacteria-killing compound, data on whose structure was captured in 2008 and stored on Monash’s myTardis repository. This data was only analysed in 2011: i.e. careful capture and preservation of data from previous experiments was key to a major discovery. Contrast this with research and teaching pipelines that focus on single end points (papers or graduates). Which leads me to:
  2. e-Research infrastructure development should follow a spiral model. For those not familiar with spiral models, they’re a process model that Barry Boehm came up with in the 1980s and they’re specifically designed to manage successive iterations of requirements gathering, risk assessment, development and planning and…
  3. The role of the University is to be the enduring home for the e-Research process.

Think about this a bit: if research output is no longer (simply) papers, but also includes data and code, what allows research to have long term value? Long term, past research maintains value because it is kept accessible by a structure of support that provides it to present researchers. This is, in Paul’s vision, the university, but its also a set of people, technologies and processes. So its the data and code repositories, its the curation effort that ensures that data is stored in accessible ways and according to meaningful schema, its the metadata that allows us to find prior work. And value for who? At the biggest picture level, society, but in a more immediate sense, value for researchers. Thus three more things:

  1. That “unknown future” is best known by people actually doing academic research. So their input in the “spiral” process is vital. In personal terms, I’m more than ever convinced that UWC needs a “e-Research Reference Group” drawn from interested academic staff from different departments that can outline requirements for future e-research infrastructure requirements.1
  2. Academics are, of course, not infrastructure builders. Infrastructure builders come in different forms – library people, IT people, etc – but in order to build effective e-Research infrastructure, they need to be partners with academics. In other words, there needs to be a common goal: research output. This is different to traditional “IT support”. In my little bubble at SANBI I’ve worked this way over the years: I’ll often partner with individuals or small groups to get work done, with them providing the “domain knowledge” and me grounding the process in computing realities (and hopefully adding a bit of software engineering wisdom etc).
  3. This partnership implies that there needs to be a growth path that recognises and rewards the work of these infrastructure-building partners.2 Paul referred to this as a “third track” in the university, distinct from both academic staff and non-academic support staff. (Ok this is a bit self-interested because I’ve been one of those “non-academic support staff (that participates in research)” for years.)

Ed’s written a blog post about the conference, and there were loads of interesting bits and pieces, such as Yvonne Sing Min’s work on building both a database (the “Vault”) and web front end to allow UCT researchers to have a central toolset for managing their research profiles (sometime similar to what we’re doing for H3ABionet with the NetCapDB), and Hein de Jager mentioning that they’re using Backblaze storage pods at UCT (gotta go see those!), and Andre le Roux’s presentation on redesigning infrastructure to accommodate research, with its focus on people, process and technology. I fear that my talk on scientific workflow systems might have been pitched at the wrong level, but it happened regardless. The presentations are online, unfortunately they don’t include the presentation from day 4 (the workshop day) yet, so Dr Musa Mhlanga’s fascinating talk on using high throughput microscopy for studying biological pathways is missing. I (and other people) tweeted a bit from the conference, using the #eresearch2013 hashtag.

Besides the talks, there was some good networking, since admins / ops people from SANBI, UWC ICS, University of Stellenbosch and UCT were all present at various times. We had a lunchtime meeting (along with Inus from CHPC) to launch a HPC Forum, which basically means that we have a mailing list and also a set of physical meetings to share experience and knowledge with regards to running High Performance Computing sites. If you’re interested in this, drop me a mail.


1. As an illustration of investing in this unknown future,  in “Where Wizards Stay Up Late: The Origins Of The Internet“, Hafner and Lyon report on J. C. R. Licklider’s request to buy a computer for BBN:[Licklider] believed the future of scientific  research was going to be linked to high-speed computers, and he thought computing was  a good field for BBN to enter. He had been at BBN for less than a year when he told Beranek he’d like to buy a computer. By way of persuasion, Lick stressed that the computer he had in mind was a very modern machine—its programs and data were punched on paper tape rather than the conventional stacks of IBM cards.

“What will it cost?” Beranek asked him.
“Around $25,000.”
“That’s a lot of money,” Beranek replied. “What are you going to do with it?”
“I don’t know.”
Licklider was convinced the company would be able to get contracts from the government to do basic research using computers. The $25,000, he assured Beranek, wouldn’t be wasted.

None of the company’s three principals knew much about computers. Beranek knew that Lick, by contrast, was almost evangelistic in his belief that computers would change not only the way people thought about problems but the way problems were solved. Beranek’s faith in Licklider won the day. “I decided it was worth the risk to spend $25,000 on an unknown machine for an unknown purpose,” Beranek said.

2. For a little rant on how hiring difficult hiring computational people to support biologists is, see C. Titus Brown’s “Dear Abby” blog post.