Plenary Session Abstract - MeerKAT: A Step Change in Big Data Management for South African Science

Abstract - In any ambitious high tech project there are dividends not fully envisaged at the outset. For example the Apollo programme have rise to Silicon Valley, CERN to the http protocol and Australian radio astronomy to Wifi. With the huge data rates of the MeerKAT telescope, not to mention the SKA when it is built, and given the transversal nature of data technologies, the dividend of modern radio astronomy will most likely be yielded in the area of Big Data. In MeerKAT approximately 2,2 Terabits of data flows from the array to the correlator, where because of fact that array is really composed of pairs of interferometers and the sum is over all pairs to obtain an image, the data rate rises to 15 terabits per second. In SKA-1 and SKA-2 the initial data rates will be in the region of 50 Terabits and 5000 Terabits respectfully, way more than any scientific instrument anywhere. These step changes in data rates necessitate new thinking on capabilities and even on policies. This discussion explores what needs to be done, in an era where often technologies move faster than the policy processes governing them do.