By Dan Bolan - University of New Hampshire
April 19, 2015
Data management intern Dan Bolan works in the rack room, the computer system hub for the ship. Image courtesy of the NOAA Office of Ocean Exploration and Research. Download larger version (jpg, 1.4 MB).
Our remotely operated vehicles (ROVs) have sensors to detect water temperature and salinity, how much oxygen is dissolved in the water, and all sorts of information on the position and orientation of the robots. The ship has its own array of sensors and sonars as well, mostly for mapping and weather conditions. In addition to oceanographic and atmospheric sensors, there are over 40 high-definition and standard-definition cameras that are mounted to the ship and ROV platforms, with new ones being added every year. Depending on the mission, we may bring down other instruments as well.
All of these sensors and cameras mean that when we use the ROVs, an expedition can generate about a dozen terabytes of information. And all of this data has to go through our system on the ship, formatting and organizing everything so that it can be used easily by scientists on shore, as they work with the shipboard scientists to plan and direct ROV dives.
To start, all of the information collected from the ROVs gets encoded as quick light pulses which are transmitted along fiber optic cables threaded inside the big steel cable that holds up Seirios. The fiber optic cables are really incredible; the part of the wire that carries the information is a very thin glass filament, only about ten micrometers wide but seven kilometers long! These flashes of light make it to our ship and are turned back into electrical signals representing our text and numbers and video. Those signals are then put onto our network.
A lot of things happen to the data streams at this point. We record the data streams to hard drives, we send real-time video and updates to the scientists on shore about Deep Discoverer's position, we make clips from the video in a very high-resolution format, and we create an eventlog to mark down all the interesting parts. (We save a copy of the full dive in a lower-quality format at the resolution it is streamed off the ship).
Scientist Andrea Quattrini enters data into the evenlog, one of the primary tools the science team uses to stay in contact with the shore-side science team and to record observations. Image courtesy of the NOAA Office of Ocean Exploration and Research, Exploring Puerto Rico’s Seamounts, Trenches, and Troughs. Download larger version (jpg, 1.4 MB).
There's a 3-second lag between when something happens and when it shows up on the video stream on OceanExplorer.noaa.gov. This is because the video has to come from the bottom of the ocean, up to space (our satellite flies in geostationary orbit more than 35,000 kilometers up), back down to Earth in New Jersey. The video is then transmitted to Maryland, is routed to Internet-2 (a high-speed network used for academic and government research), and then routed to Internet-1 (the standard Internet you use from your DSL or Cable service) via the University of Rhode Island. Then it is reformatted to make it better for streaming before it gets sent to you! The onshore scientists working with us using Internet-2, can watch with only about a four-second lag so that they can participate more effectively.
When each dive is finished, we do all the processing we need and data gets stored on the ship. We make sure that the data exists in at least two places at any given time and have duplicate servers where we keep our files. We spend quite a lot of time and energy getting this data; we'd hate to lose it!
We have a shipboard central server for the raw data and products we generate from them. This data is continually synchronized with a shoreside server so that the participating scientists will have access to the same datasets within an hour of them being collected and processed on the ship.
The really big files stay on the ship until the end of the cruise. When we disembark, we put copies of all our data on portable drive systems. These are meant to be very robust; they have five drives each and store the bits in such a way that if one of the drives breaks, you can still recover all the data from the other four regardless of which one broke.
These drives get sent to the National Center for Environmental Information archives at various locations around the country. Eventually the larger source video files will be uploaded to the CLASS system, where it will be curated by professional archivists for long-term storage. The lower-resolution video, our still images, and the dive reports get uploaded to the NOAA Central Library for open access.
If you would like the high-resolution data, you can request that as well. Because the files are too big to download easily, however, we have to send it over SneakerNet. This is where you send us a hard drive (in a sneaker box if you'd like, hence the name...we prefer a more study case, however) and we'll send it back to you with all the requested data loaded onto it. As low-tech as it sounds, this is the most effective way to send huge files, and it will be for quite a while.
