Reconstituting the Stasi files

The Stasi was the Ministry for State Security of East Germany, a feared, hated organization that kept tabs on virtually anyone and everyone in that country before the fall of the Berlin Wall and the reunification of Germany.

Prior to the final collapse of the East German regime, the Stasi tried to destroy the most critical of its secret files. They're now being reconstituted by a remarkable development in technology. The Times reports:

As the East German state crumbled in 1989, Erich Mielke, the chief of the secret police, better known as the Stasi, ordered his minions to destroy the most incriminating files, the hard evidence of a state founded on fear, spying, blackmail and betrayal.

The task of destruction was monumental. The Stasi employed 91,000 agents as well as thousands more informants to spy on friends, neighbours, fellow workers and family members — a brutal bureaucracy that produced a staggering quantity of paperwork.

The Stasi’s flimsy electric shredders (Reisswolfs: literally rip-wolves) collapsed under the strain, so the secret police continued the job by hand, working around the clock for three months. An astonishing 45 million documents were ripped up, and stuffed into rubbish bags.

The authorities had planned to burn the paper in a remote quarry, but in the chaotic final days of the communist regime it proved impossible to assemble enough trucks for the journey. So the 16,000 bags were stored in the basement of the Stasi headquarters in Berlin.

The reunified German state insisted that the files be reconstructed, and a team of 30 in Nuremberg set about manually sticking the documents together using old-fashioned puzzle methods, tweezers and lots of sticky tape. But in the course of 15 years, just 350 sackfuls of paper have been reassembled this way: completing the task, it was estimated, would take at least another 400 years.

In 1996 Dr Nickolay, an expert in image processing at the Fraunhofer Institute for Production Facilities and Construction Technology in Berlin, saw a television programme about the destroyed files and set about building a prototype machine that could do the job automatically. The result is the E-puzzler, the most sophisticated digital pattern-recognition system in the world.

In the Fraunhofer office in Berlin, Dr Nickolay displays a demonstration model of the invention which, he says, should complete the task of reconstructing all the ripped files in a matter of years. The German Government gave the go-ahead, and funding, for the first 400 bags to be reconstructed. In some ways the E-puzzler works like a human doing a jigsaw, only much faster and without the benefit of a box-lid to show what the puzzle should look like.

First, the fragments from each bag are smoothed out and fed into a large scanner: not just ordinary paper but carbon paper, photographs, microfilm, newsprint and folders. The unique characteristics of each piece — shape, colour, font, texture, handwriting, paper-type, edges and thickness — are stored digitally. Using an algorithm, the computer groups together similar fragments to reduce the “search space”, and then locates pieces that join up by matching the different characteristics.

The task was made slightly easier by the fact that the Stasi rippers tended to bundle the scraps from sets of files into a single bag.

Dr Nickolay, a cheery and rotund figure who could hardly look less like a historical avenger, hands me a box full of torn pieces of paper, some handwritten, some typewritten, some several inches wide but some smaller than postage stamps. These have already been scanned front and back. On the first computer monitor they appear as a disorganised jumble, but when the machine is activated they begin to dance around the screen as various combinations are tried: within a second, the document is assembled on the adjoining monitor.

“I never really liked puzzles,” he says ruefully. “Not Su-Doku, not crosswords. Now, I dream jigsaw puzzles.”

Technicians using his machine have already reconstructed some 40 bags of torn documents. With each bag containing around 5,000 pages, and each page torn into at least 15 pieces, this alone represents a three million-piece jigsaw. With enough funding, an industrial-scale E-puzzler could process 10,000 sheets an hour.

The paranoid East German state spied on its people on a scale unequalled in history, with as many as one informer for every seven citizens. The documents the Stasi tried to destroy are only a fraction — perhaps 5 per cent — of the secret police archive, but they represent the most explosive evidence of Stasi malfeasance.

“These documents were torn because they were important,” says Dr Nickolay. “They were also still active in 1989, which means the investigations were still ongoing.” The BStU, the federal commission for state security files, is in overall charge of the Stasi archive, and access to the reconstructed documents, some of which contain sensitive information on living people, will be tightly controlled.

The East German police state — memorably evoked in the 2006 film The Lives of Others — remains an object of grim fascination for both Germany and the world. The BStU receives 8,000 requests a year from Germans anxious to uncover the past.

The reconstructed files have already shed light on some of the Stasi’s darkest activities, including internal repression and espionage against the West, the use of drugs in sport and attempts to cover up a train crash in which 75 children were killed.

But the technology promises to do more than the historical equivalent of reconstructive surgery. Museum curators and archaeologists have used it to help to reconstruct broken terracotta figures from China and papyrus documents from Iraq. Next month the files from the Jewish archive in Buenos Aires, badly damaged in a terrorist attack in 1984, will be brought to Berlin, where Fraunhofer technicians will begin the task of salvage and reconstruction. The equipment has played a role in a German tax fraud case.

Such is the power of the technology that, as well as matching up paper fragments torn up by hand, it can also analyse material that has been through a shredder, although the process is slower. Even documents that have shredded twice and resemble confetti can be digitally reassembled.

. . .

The East German secret police destroyed countless lives, and then tried to destroy the evidence. Their files tell of bizarre political culture, peering secretly into the lives of others. Today that secrecy is being broken down, one page at a time, and their own activities fully probed, thanks to a machine that is making history.

There's more at the link.

It's fascinating to think that new technology can piece together even such tiny fragments . . . but also rather worrying. I keep a shredder at home to destroy all confidential documentation, for fear of identity theft using the contents of my garbage bags. How long before such technology becomes widespread, so that even common-or-garden thieves can make use of it to nullify my precautions? Does this mean we're going to have to go to a burn barrel in the back yard, to reduce shredded waste to ashes?

Peter