The Department of Defense, like many companies and organizations, have leveraged the advances in supercomputing to compute increasingly complex and large computational problems. File sizes have equally expanded with the growth in computing power, especially if the files contain very large sets of data. It’s not uncommon for files to occupy tens or hundreds of gigabytes, and transferring these files between supercomputing centers through standard means could take hours, or even days.
The need to transfer large amounts of data quickly led to the development of parallel file transfer software packages, which use multiple paths between computers to send the information. The military recommends the program Multiple Path Secure.
Copy (MPSCP) for transferring data between its supercomputing centers; however, this software package did not meet all its needs. They tasked the experts at the Ohio Supercomputer Center to improve MPSCP by improving user functions and updating the software manuals and, most importantly, adding the ability to encrypt sensitive information — without substantially sacrificing transfer speeds.
“The version of MPSCP we created not only was significantly easier to use, it also could encrypt and transfer data much faster when using more than one data stream,” said Brian Guilfoos, a computational science researcher at OSC and lead developer on the project.
The OSC software development team evaluated the performance of its modified MPSCP by recording total transfer times for a directory containing 10 files, each one gigabyte in size, for different numbers of data streams and with encryption enabled or disabled. They also compared the amount of time it took for the same transfer using secure copy, which encrypts transferred data, making it functionally similar to the encryption-enabled version of MPSCP.
“As expected, encrypted file transfers are slower than non-encrypted, but multiple paths still provide a significant improvement over secure copy, even when transferring encrypted information,” Guilfoos said.
--
Project lead: Jose Unpingco, Ph.D., OSC
Research title: Sharing & mining SIP (Signal Image Processing) data
Funding source: Department of Defense High Performance Computing Modernization Program
