New ‘chaos engineering’ tool shared between DOD software factories

The Air Force’s Kessel Run software factory is transitioning to the Navy a tool that it has been developing for the past two years that is designed to emulate persistent enemy attacks on a system.

The Navy’s Black Pearl software factory will be the first group outside of Kessel Run to get the tech stack and list of best practices on implementing it. But eventually, the goal is for as many coders to get their hands on it as possible, lead engineer Omar Marrero told FedScoop.

The tech stack and Air Force team behind it are jointly known as Bowcaster, named after the weapon Star Wars character Chewbacca used in the film series. And the discipline behind their work is something referred to as chaos engineering.

“You have to constantly break the system to find where our weaknesses are,” said Marrero, whose official title is chaos and performance engineering lead. “That’s essentially what chaos engineering is.”

The idea behind chaos engineering is to unleash unpredictable, persistent attacks that can still be controlled in what exactly they target within a system to emulate an enemy. Kessel Run launched its first internal attack using the system in the summer of 2020 after launching the program in 2019.

Marrero said the idea to put resources into chaos engineering came organically from the need to more thoroughly test systems. He said he attended several tech conferences to learn from others that had deployed similar systems, even though he already has a background in this type of cybersecurity testing.

“As part of my career in the Air Force I have always done some flavor of chaos,” Marrero said in an interview.

The lessons the Air Force learned from others and in its own practice developing the tech stack is part of what it will be transitioning to Black Pearl as part of a chaos engineering “playbook.” It will also be porting code into Platform One’s software repository Iron Bank for others to start experimenting with.

One of the biggest lessons Marrero and the team learned was to “control the blast radius,” meaning don’t let the code start unplugging too many things.

Sharing tech stacks and tools like Bowcaster is a practice Kessel Run plans to continue. The Air Force and Navy are working on a new memorandum to share even more code between the two services.