IceCube Winterover Training

I have been in Madison, WI for training for almost three weeks.  I am training to be what we call a Winterover.  This name comes from the fact that I will be spending a full year at the South Pole.  Specifically, I will be spending the winter there.  During the Antarctic summer, planes go in and out of the South Pole regularly, and by regularly, I mean at least once a week or more.  During the Antarctic winter, however, planes rarely if every fly to the South Pole.  As I understand it, they have actually only ever flown in three times during the winter, and that was for big emergency situations.  In general, once the last plane leaves for the season, you are stuck there for 6 months with almost no hope of getting out!  For that reason, you have to go through all the medical and psychological evaluations that I talked about in a previous post.

All that being said, my training now, is mostly on the technical side.  Many people think of this as a scientific position, and to some degree it is.  But the work I am going to be doing at the pole is mostly network and system administration.  The IceCube Neutrino observatory is a large experiment that collects a great deal of data.  There is so much data that we can't even get it all to the North through our satellite connections.  Instead, we have a small cluster of servers located at the South Pole that collects and processes the data to select out only the most interesting parts.  All of the data are saved and eventually shipped to the North on an airplane, but the most interesting data are sent through a satellite connection.  While I am down there, I will be working to fix any problems that arise in the computers or network that collects the data and does the processing.

For the first couple of weeks, my training was largely software based.  There are many different custom applications that process the data collected by the detector.   Even the hardware that collects the data has to be watched and occasionally fixed or upgraded.  There is software that converts all the data collected to useful information, software that filters out the stuff we don't care about, software that does calibration, and even software that moves and saves information and files to the right places whether it is to the satellite or to disks to be sent north later.  As a winterover, I have to know how these systems work and how to fix them if things break.  This could be as simple as rebooting a server or as complicated as reading through tons of log files to try to figure out problems that don't make any sense!  Thankfully, the systems are fairly well designed so it is usually not too much work to fix problems, and I should always have some support from the IceCube members up in the North to help with any really big problems.  Though, in the end, it is the winterovers who have to implement most of these solutions, which is why there is so much training.

This past week I have also been doing some network and system admin training.  I have a background in network administration, so I am not seeing anything too far outside of my experience.  The systems on this network are more advanced than the systems I have used in the past and there is way more thought put into the network as a whole, but the general concepts are pretty much the same.  That being said, I think this would be very difficult and confusing if I didn't have at least somewhat of a networking background.

If anyone was interested in one day working as a winterover for IceCube in the future, I would strongly recommend spending time learning Linux and general network and system administration.  These are the skills that seem to be the most valuable so far.  Spend time breaking and fixing computers.  Installing all sorts of software that other Linux users and system admins use from day to day and try to learn how to use it in interesting ways.  It doesn't hurt to have a PhD in physics, but it really helps to have a solid understanding of bash! (If you don't know what bash is, that might be a good place to start!) I am excited to be a part of IceCube in this capacity and to  see how things work from this perspective, and I am really looking forward to my trip down to the South Pole!