Why 200 diehard coders were in a mad scramble to rescue NASA’s climate data

On February 11th, 200 elite coders and software engineers were crammed in the Doe Library at the University of California, Berkeley, racing to download NASA’s Earth science data in a hackathon. Equipped with laptops, whiteboards, and coffee, the group’s goal was to rescue data that may be deleted or hidden under President Trump’s administration, according to Wired. 

An animation showing how carbon dioxide moves around the planet. Image source: NASA.

The process involved developing web-crawler scripts to search the internet in search of federal data and patching it together into coherent data sets. The hackers were also keeping track of data as it disappeared. For example, the Global Data Center’s reports and one of NASA’s atmospheric carbon dioxide data sets has already been removed from the web.

To keep data organized, the hackers split into two groups: taggers and baggers. The taggers loosed web crawlers to make sure that sites made it into the Wayback Machine and tagged pages, sites, and data sets too large for archive.org. Baggers then took those data sets and banged out custom scripts in order to carefully comb out the data from the government web presence.

Of course, none of this is simple.

“All of these systems were written piecemeal over the course of 30 years,” said chief technology officer at UtilityAPI and volunteer for the Berkeley bagger group, Daniel Roesler. “There’s no coherent philosophy to providing data on these websites.”

Another coder, known as Tek, ran into a wall trying to download multi-satellite precipitation data from NASA’s Goddard Space Flight Center. Starting in August, access to Goddard Earth Science Data required users to log in. But with a bit of legal digging around, Tek found a buried link to the old FTP server, so he began downloading. By the end of the day, he had data for all of 2016 and some of 2015. It took at least another 24 hours to finish.

The non-coders hit dead ends, too. Throughout the morning, they racked up “404 Page not found” errors across NASA’s Earth Observing System website. More than once, they ran across databases that had already been emptied out, such as the Global Change Data Center’s reports archive and one of NASA’s atmospheric CO₂  data sets.

But this is where the real problem lies because the group of hackers isn’t sure when exactly some of this data disappeared or if anyone backed it up first. 

The Trump administration's stance, according to former Congressman Robert Walker, is that “Earth-centric science is better placed at other agencies where it is their prime mission.” Late last year, he said that NASA should instead focus its resources on deep-space exploration.

Walker served in the House of Representatives from 1977 to 1997, and was the chairman of the House Committee on Science for the final two years of his tenure. Since retiring, he’s worked as a lobbyist and served on a Department of Energy commission on hydrogen fuel cells.

“Climate research is necessary but it has been heavily politicized, which has undermined a lot of the work that researchers have been doing,” Walker said in November 2016. “Mr. Trump’s decisions will be based upon solid science, not politicized science.”

Fortunately, by the end of the day when the hackathon at the University of California, Berkeley concluded, the coders successfully downloaded thousands of pages — essentially all of NASA's climate data — onto the Internet Archive, a digital library.

Mission accomplished.