Puzzle History: Codebreaking and the NSA, part 3

[Image courtesy of NSA’s official Twitter account.]

At the end of part 2 in our series, we left off during the early days of the NSA, as American cryptographers continued to labor under the shadow of the Black Friday change in Russian codes.

You may have noticed that part 2 got a little farther from puzzly topics than part 1, and there’s a reason for that. As the NSA evolved and grew, codebreaking was downplayed in favor of data acquisition. The reasons for this were twofold:

1. Context. You need to understand why given encrypted information is important in order to put it toward the best possible use. As Budiansky stated in part 1, “The top translators at Bletchley were intelligence officers first, who sifted myriad pieces to
assemble an insightful whole.”

2. Russian surveillance and bugging continued to grow more clever and sophisticated, pushing attention away from codebreaking. After all, what good is breaking codes or developing new ones if they can just steal unencrypted intel firsthand by monitoring
agents in the field?

Moving forward, the NSA would continue to pursue all manner of data mining, eventually leaving behind much of the codebreaking and analysis that originally formed the backbone of the organization. But that was in years to come. Cryptography was still a major player in NSA operations from the ’50s and onward.

[The progression of “secret” and “top secret” code words.
Image courtesy of NSA’s official Twitter account.]

In May 1956, NSA cryptanalytic veterans pushed a proposal titled “Recommendations for a Full-Scale Attack on the Russian High-Level Systems,” believing that specially designed computers from IBM could provide the key for cracking the impenetrable Russian cryptography wall. Some cryptographers believed that ever-increasing processor speeds would eventually outpace even sophisticated codes.

By 1960, the NSA had spent $100 million on computers and analytical tools.

The problem? The NSA was collecting so much information that their increasingly small team of cryptoanalysts couldn’t dream of processing even a tiny portion of it.

But the quest for data access would only grow more ambitious.

In the wake of Sputnik’s launch in October of 1957, US signals intelligence would go where no man had gone before. The satellite GRAB, launched alongside Transit II-A in June of 1960, was supposedly meant to study cosmic radiation. (GRAB stood for Galactic Radiation and Background.)

[Image courtesy of NSA’s official Twitter account.]

But it was actually intended to collect radar signals from two Soviet air-defense systems. This was the next step of ELINT, electronic intelligence work. (The younger brother of SIGINT.)

The NSA would later find a huge supporter in President Lyndon Johnson, as the president was heavily invested in SIGINT, ELINT, and any other INTs he could access. This did little to quell the intelligence-gathering rivalry growing between the CIA and NSA.

Of course, that’s not to say that the NSA ceased to do any worthwhile work in codebreaking. Far from it, actually.

During the Vietnam War, NSA analysts pored over North Vietnamese signals, trying to uncover how enemy pilots managed to scramble and respond so quickly to many of the US’s airstrikes conducted during Operation Rolling Thunder.

Careful analysis revealed an aberrant character (in Morse code) in messages that appeared in North Vietnamese transmissions before 90 percent of the Rolling Thunder airstrikes. By identifying when the enemy used that aberrant character, the analysts
were able to warn US pilots whether they were heading toward a prepared enemy or an unsuspecting one during a given sortie.

Other NSA teams worked to protect US communications by playing the role of an enemy analyst. They would try to break US message encryptions and see how much they could learn from intercepted US signals. Identifying flaws in their own procedures — as well as members of the military who were cutting corners when it came to secured communications — helped to make US communications more secure.

[Image courtesy of NSA.gov.]

In 1979, Jack Gurin, the NSA’s Chief of Language Research, wrote an article in the NSA’s in-house publication Cryptolog, entitled “Let’s Not Forget Our Cryptologic Mission.” He believed much of the work done at the agency, and many of the people
hired, had strayed from the organization’s core mission.

The continued push for data acquisition over codebreaking analysis in the NSA led to other organizations picking up the slack. The FBI used (and continues to use) codebreakers and forensic accountants when dealing with encrypted logs from criminal organizations covering up money laundering, embezzlement, and other illegal activities.

And groups outside the government also made impressive gains in the field of encryption, among them IBM’s Thomas J. Watson Research Center, the Center for International Security and Arms Control, and even graduate student programs at universities like MIT and Stanford.

For instance, cryptographer Whitfield Diffie developed the concept of the asymmetric cipher. Joichi Ito explains it well in Whiplash:

Unlike any previously known code, asymmetric ciphers do not require the sender and receiver to have the same key. Instead, the sender (Alice) gives her public key to Bob, and Bob uses it to encrypt a message to Alice. She decrypts it using her private key. It no longer matters if Eve (who’s eavesdropping on their conversation) also has Alice’s public key, because the only thing she’ll be able to do with it is encrypt a message that only Alice can read.

This would lead to a team at MIT developing RSA, a technique that implemented Diffie’s asymmetric cipher concept. (It’s worth noting that RSA encryption is still used to this day.)

[Image courtesy of Campus Safety Magazine.com.]

The last big sea change in encryption came when the government and military realized they no longer had a monopoly on codebreaking technology. Increased reliance and awareness of the importance of computer programming, greater access to computers with impressive processing power, and a groundswell of support for privacy from prying government eyes, led to dual arms races: encryption and acquisition.

And this brings us to the modern day. The revelations wrought by Edward Snowden’s leak of NSA information revealed the incredible depth of government data mining and acquistion, leading some pundits to claim that the NSA is “the only part of government that actually listens.”

Whatever your feelings on Snowden’s actions or government surveillance, there is no doubt that the National Security Agency has grown and changed a great deal since the days of cracking the ENIGMA code or working with the crew at Bletchley Park.

Where will American codebreaking go next? Who knows? Perhaps quantum computing will bring codes so complicated they’ll be impenetrable.

All I know is… it’s part of puzzle history.


I hope you enjoyed this multi-part series on the history of 20th-century codebreaking in America. If you’d like to learn more, you can check out some of the valuable sources I consulted while working on these posts:

Code Warriors: NSA’s Codebreakers and the Secret Intelligence War Against the Soviet Union by Stephen Budiansky

Whiplash: How to Survive Our Faster Future by Joichi Ito

The Secret Lives of Codebreakers by Sinclair McKay


Thanks for visiting PuzzleNation Blog today! Be sure to sign up for our newsletter to stay up-to-date on everything PuzzleNation!

You can also share your pictures with us on Instagram, friend us on Facebook, check us out on TwitterPinterest, and Tumblr, and explore the always-expanding library of PuzzleNation apps and games on our website!

Puzzle History: Codebreaking and the NSA, part 2

[Image courtesy of NSA’s official Twitter account.]

At the end of part 1 of our look at the history of the NSA and American codebreaking, we left off with the pivotal Black Friday event.

On November 1, 1948, all intel coming from monitored Soviet signals went quiet. All traffic on military, naval, and police radio links was replaced with dummy messages. It was such an unprecedented and alarming event that London and Washington briefly considered that it might’ve been the first indication of preparations for war.

According to Code Warriors author Stephen Budiansky:

The full extent of the disaster only became apparent the following spring when real traffic started reappearing on the radio nets, now employing greatly improved — and completely unbreakable — technical and security procedures. The keying errors or other mistakes that had allowed most of the Soviets’ machine-enciphered military traffic to be routinely read by US and British codebreakers for the last several years had been corrected, and the much more disciplined systems that now replaced them slammed the cryptanalytic door shut.

Even the one-time pads that had offered some hope to attentive American codebreakers were updated, eliminating the ability to sort messages by which organization they originated from.

Codemakers had suddenly outpaced codebreakers.

[The Kryptos sculpture outside CIA Headquarters. The NSA cracked
several of its codes before the CIA did. Image courtesy of Slate.com.]

The Office of Naval Intelligence wanted to take over from Signals Intelligence (SIGINT), demanding to see “everything” so they could do the job. They claimed SIGINT should limit their work to message translation, leaving interpretation to “the real experts.” This sort of territorial gamesmanship would continue to hamper government organizations for decades to come.

And that demand to see everything? That probably sounds familiar, in light of the revelations about government data collection and the PRISM program that were revealed in Edward Snowden’s leaks.

Black Friday was the start of all that, a shift from codecracking to the massive data collection and sifting operation that characterized the NSA for decades to come.

More amazingly, there was SO MUCH information collected during World War II that SIGINT was still poring over it all in 1949, decrypting what they could to reveal Soviet agents in the U.S. and England.

The fact that a high-ranking member of British Intelligence at the time, Kim Philby, was actually a Soviet double agent complicated things. After a decade under suspicion, Philby would flee to the Soviet Union in 1963, stunning many friends and colleagues who had believed in his innocence.

[The spy and defector, honored with a Soviet stamp.
Image courtesy of Britannica.com.]

Although the Russians had flummoxed SIGINT, other countries weren’t so lucky. The East German police continued to use ENIGMA codes as late as 1956. Many of the early successes in the Korean War were tied to important decryption and analysis work by SIGINT. Those successes slowed in July of 1951, when North Korea began mimicking Russia’s radio procedures, making it much harder to gain access to North Korean intel.

Finally, the chaotic scramble for control over signal-based data gathering and codebreaking between the government and the military resulted in the birth of the National Security Agency on November 4, 1952, by order of President Truman.

One of the first things the NSA did? Reclassify all sorts of material involving historical codebreaking, including books and papers dating back to the Civil War and even the American Revolution.

[The actual report that recommended the creation of the NSA.
Image courtesy of NSA’s official Twitter account.]

The creation of the NSA had finally, for a time at least, settled the issue of who was running the codebreaking and signals intelligence operation for the United States. And they were doing fine work refining the art of encryption, thanks to the work of minds like mathematician and cryptographer Claude Shannon.

One of Shannon’s insights was the inherent redundancy that is built into written language. Think of the rules of spelling, of syntax, of logical sentence progression. Those rules define the ways that letters are combined to form words (and those words form sentences, and those sentences form paragraphs, and so on).

The result? Well, if you know the end goal of the encoded string of characters is a functioning sentence in a given language, that helps narrow down the amount of possible information contained in that string. For instance, a pair of characters can’t be ANYTHING, because letter combinations like TD, ED, LY, OU, and ING are common, while combos like XR, QA, and BG are rare or impossible.

By programming codecracking computers to recognize some of these rules, analysts were developing the next generation of codebreakers.

Unfortunately, the Russian line was holding. The NSA’s failure to read much, if any, Soviet encrypted traffic since Black Friday was obviously becoming more than just a temporary setback.

Something fundamental had changed in the nature of the Russian cryptographic systems, and in the eyes of some scientific experts called in to assess the situation, the NSA had failed to keep up with the times.


I hope you’re enjoying this look at the early days of America’s 20th-century codebreaking efforts. Part 3 will continue next week, with the sea change from active codebreaking to data mining, plus Vietnam, the space race, and more!


Thanks for visiting PuzzleNation Blog today! Be sure to sign up for our newsletter to stay up-to-date on everything PuzzleNation!

You can also share your pictures with us on Instagram, friend us on Facebook, check us out on TwitterPinterest, and Tumblr, and explore the always-expanding library of PuzzleNation apps and games on our website!

Puzzle History: Codebreaking and the NSA

ohr577n

[Image courtesy of NSA.gov.]

The National Security Agency has been in the news a lot over the last few years.

It arguably all started with Edward Snowden and the revelations about government surveillance, both domestic and foreign, that emerged in his wake. Between President Trump’s intimations of Obama-era wiretapping (which also supposedly involved England’s GCHQ) and recent news stories about NSA contractor Reality Winner leaking information, the NSA continues to draw mainstream attention in the 24-hour news cycle.

When you factor in the declassification of codebreaking intel during and after World War II, we know more about the NSA’s inner workings than ever before.

You might be asking what the NSA has to do with puzzles. Well, everything. Because the NSA was born as a codecracking organization.

enigma

The NSA was founded in November of 1952, but its formative stages began during World War II, as codebreakers were recruited in the U.S. starting in 1943. Not only were they tasked with tackling the German ENIGMA code, but their secondary mission was to solve “the Russian problem.” This group was known as Signals Intelligence, or SIGINT.

William Friedman, one of the early figures in American codebreaking, described cryptanalysis as “a unique profession, demanding a peculiar king of puzzle-solving mentality combined with patience. So staffing this new organization was a curious endeavor.”

Those who were recruited came from all walks of life:

Career officers and new draftees, young women math majors just out of Smith or Vassar, partners of white-shoe New York law firms, electrical engineers from MIT, the entire ship’s band from the battleship California after it was torpedoed by the Japanese in the attack on Pearl Harbor, winners of puzzle competitions, radio hobbyists, farm boys from Wisconsin, world-traveling ex-missionaries, and one of the World’s foremost experts on the cuneiform tablets of ancient Assyria.

04/11/14 imitation game - bletchley park, milton keynes

A large campus was built that echoed the style and efforts of Britain’s Bletchley Park, including Alan Turing’s calculating machines, the bombes. Efforts on both sides of the Atlantic centered on cracking ENIGMA, the German codes used in all sorts of high-level communications. The teams worked alongside the bombes to try to determine which of the 456, 976 possible codes was being used in a given piece of communication.

It was a truly Herculean effort.

But while nearly half the staff focused on the Germans, others focused on cracking Russian codebooks, where words were translated into four-digit codes. Often, decrypting these codes involved “brute force” efforts, poring through numerous messages to pair up messages that used similar numerical groups, meaning they used the same cipher.

This would only work if the Soviets were lazy in their production of so-called “one-time pads,” encryption devices that had a particular code, which would be used once and then thrown away. Brute force codebreaking revealed that some of the one-time pads had been used more than once, a lapse in Soviet security that could work to the advantage of U.S. intelligence.

That deduction led to another stunning discovery: cracking the system used in encrypted messages to tell agents which encryption was used in a given missive. You see, each encoded message contained within it a code that dictated the cipher necessary to decrypt the message.

The Russians would later complicate this work by employing multiplexers: devices that would transmit numerous messages at once, making it harder to separate one message from another in the same dispatch.

hqdefault6

[Image courtesy of Virtantiq.com.]

The Germans would unwittingly aid the US in their Russian codebreaking efforts when a POW camp in Bad Aibling, Germany, was captured by the US army, and they uncovered a German device designed to “de-multiplex” Russian messages. The device was called the HMFS, because Hartmehrfachfernschreiber, while a great deal of fun to type, is hard to say quickly.

After World War II ended, U.S. intelligence consolidated their efforts on “the Russian problem,” continuing their work unraveling the Russian codebooks. Slowly, the codemasters began determining which organizations in the Soviet government used which codes. Even if the codes weren’t broken yet, it helped the intelligence community organize and prioritize their efforts.

The problem? They had a very tight timeframe to work in. Those duplicated codebooks were produced during a very small window of time in 1942, and only issued to Soviet agents in the three years that followed. By 1947, SIGINT analysts knew the Soviets would soon run out of the duplicated pads. Once they did, those recurring patterns of encrypted numbers would stop, and the best chance for cracking the Soviet codes would be lost.

Still, there was reason to be encouraged. Some important code words had been identified. TYRE was New York City, SIDON was London, and CARTHAGE was Washington; ENORMOZ appeared often enough that they determined it referred to atomic bomb research in Los Alamos.

It would also be revealed, through careful analysis of decrypted intel, that Soviet agents were embedded in both the U.S. Justice Department and in England’s Bletchley Park campus. The Justice Department agent was identified and tried, but released after the court found insufficient evidence to place her under surveillance in the first place.

This was one consequence of the secrecy surrounding codebreaking: an unwillingness to reveal their codebreaking success by turning over evidence of it. (As for the Bletchley Park spies, one was identified in 1951 and confessed in 1964. The other was never identified.)

By this time, the Russians had gone over ENIGMA machines captured during the German retreat, and had unraveled not only how the devices worked, but how to improve upon them. This would lead to the next-generation Russian Fialka machine.

fialka

With ever-increasing complexity when it came to encryption, thanks to increased automation, codebreaking evolved into not just intelligence work, but intelligence analysis. After all, if you don’t know something is important, you don’t necessarily give it the attention it deserves. As researcher Stephen Budiansky put it, “The top translators at Bletchley were intelligence officers first, who sifted myriad pieces to assemble an insightful whole.”

It also led to bigger, faster machines, like Goldberg and Demon, two computation machines designed to more efficiently pore over the vast amount of encrypted information being intercepted by the various U.S. intelligence services.

In 1948, though, the game changed. It changed so dramatically that November 1, 1948, is still remembered in NSA circles as Black Friday.


I hope you’re enjoying this look at the early days of America’s codebreaking efforts. Part 2 will continue next week, with a look at the rise of the NSA, Cold War cryptography, and more!

[Quotes and certain photos were sourced from Code Warriors: NSA’s Codebreakers and the Secret Intelligence War Against the Soviet Union by Stephen Budiansky.]


Thanks for visiting PuzzleNation Blog today! Be sure to sign up for our newsletter to stay up-to-date on everything PuzzleNation!

You can also share your pictures with us on Instagram, friend us on Facebook, check us out on TwitterPinterest, and Tumblr, and explore the always-expanding library of PuzzleNation apps and games on our website!

The GCHQ Strikes Again!

Last year, one of the toughest puzzles I encountered all year was the GCHQ Christmas Card.

The GCHQ — Government Communications Headquarters — provides security and intelligence services for the British government. Back when they were known as GC&CS — Government Code and Cypher School — they were responsible for funding Bletchley Park and its successes cracking the German “Enigma” code during World War II.

And last Christmas, they released a puzzly Christmas card that challenged the staunchest puzzlers, with over 600,000 submissions, but only 3 successful solutions!

This year, they’re doing things a little bit differently.

[Click here for a larger version.]

This puzzle is the first step in a larger event that the GCHQ expects will take MONTHS to solve. (Their official due date for submissions is February 28, 2017!)

The big change here is that instead of a series of webpages available free to the public, they’re releasing this puzzly master challenge as a book containing over 140 codes and puzzles.

But it’s for a good cause, as “all GCHQ’s proceeds from sales of the book will be donated to the Heads Together campaign, supported by HRH The Duchess of Cambridge, who wrote the foreword for The GCHQ Puzzle Book.”

The Duchess writes:

I have always been immensely proud of my grandmother, Valerie Glassborow, who worked at Bletchley Park during the Second World War. She and her twin sister, Mary, served with thousands of other young women as part of the great Allied effort to break enemy codes. They hardly ever talked about their wartime service, but we now know just how important the men and women of Bletchley Park were, as they tackled some of the hardest problems facing the country.

In a new century, their successors at GCHQ continue this intellectual tradition. Like their Bletchley predecessors, they have become well known for valuing and understanding the importance of mental wellbeing. This is so important when dealing with such discretion and the pressure which comes with this.

William, Harry and I are very grateful that this book is supporting our Heads Together Campaign. I hope it will not only amuse and challenge readers, but help to promote an open discussion of mental health problems, which can affect anyone, regardless of age or background. Together, we are aiming to change the national conversation around mental health from stigma and fear to openness and understanding. Those who buy this book and support the Heads Together campaign will be playing a part in helping people get the important mental health care they deserve.

Puzzles and charitable works: a perfect holiday match, to be sure.

For US shoppers, the book is available as a Kindle ebook through Amazon.com, but if you want a paperback copy, you’re better off ordering it from Amazon.co.uk. With shipping and conversion, it still cost me less than $20.


And thanks for visiting PuzzleNation Blog today! Be sure to sign up for our newsletter to stay up-to-date on everything PuzzleNation!

You can also share your pictures with us on Instagram, friend us on Facebook, check us out on TwitterPinterest, and Tumblr, and explore the always-expanding library of PuzzleNation apps and games on our website!

A puzzly British Christmas card!

One government agency in England celebrates Christmas a little bit differently than most.

The GCHQ — Government Communications Headquarters — provides security and intelligence services for the British government. Back when they were known as GC&CS — Government Code and Cypher School — they were responsible for funding the Bletchley Park successes cracking the German “Enigma” code during World War II.

And for Christmas this year, they’ve released a puzzly Christmas card that’s sure to challenge even the staunchest puzzlers.

Step 1 of the puzzle is a logic art puzzle where you have to deduce where to place black squares on an open grid in order to form a picture.

Each column and row has a series of numbers in it. These numbers represent runs of black squares in a row, so a 1 means there’s one black square followed by a blank square on either side and a 7 means 7 black squares together with a blank square on either side.

Once you’ve solved this puzzle, you can use it to unlock the next puzzle in the chain.

From an article on GCHQ.gov.uk:

Once all stages have been unlocked and completed successfully, players are invited to submit their answer via a given GCHQ email address by 31 January 2016. The winner will then be drawn from all the successful entries and notified soon after.

Players are invited to make a donation to the National Society for the Prevention of Cruelty to Children, if they have enjoyed the puzzle.

This is one majorly challenging Christmas card. After you’ve conquered the logic art puzzle, you’ll confront brain teasers, palindromes, pattern-matching, deduction, number progressions, codebreaking, cryptic crossword-style cluing, and more.

I would highly recommend teaming up with another puzzle-minded friend (or more) and trying your luck. Let us know how far you get! (And you can hit up this article from the Telegraph for aid as well.)


Thanks for visiting PuzzleNation Blog today! Be sure to sign up for our newsletter to stay up-to-date on everything PuzzleNation!

You can also share your pictures with us on Instagram, friend us on Facebook, check us out on TwitterPinterest, and Tumblr, and explore the always-expanding library of PuzzleNation apps and games on our website!