The ‘magic’ of continuous IT service delivery

“There are two things I care about a lot,” said Mark Schwartz, chief information officer of the Department of Homeland Security’s U.S. Citizenship and Immigration Services division. “One is – waste is bad. The second is – responsiveness is good.” And as Schwartz sees it, “Continuous delivery is the obvious choice for how to do things if you care about those two things.”

Continuous delivery is emerging as the preferred way to think about developing software. While many organizations have come to embrace the logic of coding in short, agile sprints, the need to keep adapting to changing customer demands has given rise to a more continuous and collaborative approach to delivering IT services. Continuous delivery builds on the disciplines of agile development.

While agile development strikes many as an oxymoron in the world of government, the concept has nonetheless taken root at a growing number of federal agencies and gained momentum faster than many might suspect.

“It’s unbelievable how fast we are changing things in the government,” Schwartz said at a government and industry technology forum this week, sponsored by the American Council for Technology and Industry Advisory Council.

“A lot of the conversation today is about how you get from three-month release cycles to daily-release cycles. A year ago, we were arguing about whether you should have six-year release cycles or be agile and have three-month release cycles.”

Mark Schwartz, CIO, U.S. Citizenship and Immigration Services, leading exercise on implementing continuous delivery at 2015 ACT-IAC conference, (FedScoop)

The contrast between the way agencies used to develop software and what Citizenship and Immigration Services, the FBI, the U.S. Census Bureau, the Food and Drug Administration, and others are doing it today are as striking as their outcomes.

In the old model, before continuous deliver took effect, said Schwartz, changing fields on a Homeland Security Web page meant creating a program team, putting in a governance process, going through a DHS acquisition review board, finding contractors, and developing and deploying the software that had to pass through testing and staging environments.

“When you finally get it to production, it’s eight months later. With our continuous delivery pipeline, you make your change … and boom, it’s in production and people are using it right away,” Schwartz said. Getting to “boom” of course can be a quixotic undertaking, as Jeff Johnson discovered after joining the FBI in 2009. He now serves as chief technology officer and assistant director of IT management. He still gets confronted with the FBI’s costly failure more than a decade ago to modernize its case management software system. To this day, it remains a case study on why large government IT projects are prone to fail, and why continuous delivery is making a big difference.

Johnson, who worked for Lehman Brothers and a tech engineering firm before coming to the FBI, persuaded FBI officials to turn their IT development process upside down, vowing he would take the “talent and capabilities we have … and start showing progress every two weeks.”

“I can guarantee you I can allow a group of innovators in the basement of the Hoover Building to go build things and test them faster than we can plan them,” he recalled telling them.

Persuading his bosses was only half the battle. Even his team members said, “You can’t do that. The [FBI’s inspector general] will crucify it.”

Jeff Johnson, FBI chief technology officer discusses agile development at 2015 ACT-IAC conference. (FedScoop)

A number of employees associated with the program were so skeptical about his scheme, “not only were they leaving the program, they were taking the project name off their resume.”

Johnson’s response: “They won’t crucify you. They’ll crucify me. I’ll take that risk. And here’s how: I’ll mitigate that risk. Not by writing them another report. I’m going to put my confidence in you [his team] that you can write code faster than [the IG] can write reports.”

His demand and mantra to his team was equally disruptive. “I’m not going to count any activity, anything that’s partially done. It’s only done if it’s fully implemented and the customer has seen it.”

And if the IG has “something legitimate in their draft, we will fix it before they complete the review process.” Having a functional agile development gives Johnson confidence that if he is taken to task in the review process, his team can fix things fast enough that Johnson can legitimately tell the IG, “that’s not factually accurate.”

There are many ingredients to getting an IT development team to begin developing in agile sprints. It requires leadership’s commitment, a clear strategy, buy-in from the business or mission owners to engage in regular reviews, and the involvement of acquisition, security teams, legal counsel when necessary.

But from IT’s perspective, said Schwartz, the key is to “take a very small set of requirements – potentially just one requirement at a time – and put it through a pipeline that gets it into production as quickly as possible,” then gets feedback quickly, in what builds into a virtuous cycle.

But the real magic comes from automating “almost everything,” he said.

He described a process where “developers make changes to code very quickly. They check it in. Our continuous integration software automatically knows there’s been a change, rebuilds the whole system, runs all of the automated regression tests and static code analysis to check on coding styles, security tests, section 508 compliance tests. And if the code passes all of those tests, we have automated scripts that will deploy [the updated code] into production. We have monitoring on production so we know what [the code change] is doing. It’s all magic.”

Johnson, in a light-hearted jab, disagreed with Schwartz on one point. “It’s not magic. It’s a daily continuous discipline.”

He also challenged the notion that responsive and agile design is too difficult to use for large-scale projects in government.

“We’ve shown that it can happen in government. The FBI spent $570 million between VCF and Sentinel up to 2010” for the agency’s Virtual Case File management system and another ill-fated successor system. “We started an agile two-week process in October 2010 and in July 2012, we were done.” The whole project was completed for $30 million in just 20 months, he said.

“We can do that across government,” Johnson said, “if we engage our teams, if we’re all working on delivering that value proposition and focusing on how are we going to make our government better.”

But it starts with leadership, he insisted. “As leaders, we own the risk. It’s our names that will be in the IG report. That’s fine,” he said. “You can be wrong just as many times” with a plan as you would be using agile development.

“But now you have something that’s functional, and integrated, and tested, and you can prove whether or not it works. And if it doesn’t work, change it. And you can change it faster than you can write about it!”

Avi Bender, chief technology officer for the U.S. Census Bureau, agreed. It also involves a fundamental understanding of “who we’re serving in the public,” he said. Understanding your customers is crucial to driving innovation, he added.

Like a lot of things in government, that’s a bigger task than it appears. The Census Bureau not only collects the statistical pulse of the nation, it also provides data collection services behind the scenes for many other federal agencies.

“If you’re trying to find better ways of collecting, processing and disseminating information, ultimately, you’d like to have a good API to deliver that [information] to the public,” and then reverse-engineer that all the way back to the source of the data, he said.

But IT leaders also need to think about “key catalysts for change — three or four very strategic initiatives … that connect all the dots,” he said. One strategic area Bender is focusing on in particular that supports continuous delivery is segment architecture.

“Segment architecture sounds technical, but it’s not. It actually looks at security and data and cultural issues; it requires subject matter experts … and engaging internal and external customers,” and ultimately “how we integrate systems in a much more agile approach.”

Johnson told FedScoop during a break in the discussion that moving to agile development in biweekly sprints has actually helped the FBI as a whole as its post-9/11 mission shifted, and the need for case management tools gave way to a greater need for intelligence operations.

The FBI can now process and analyze eight times the volume of intelligence documents the agency was able to handle just last fall, he said.

At the same, the automated processes now in place to handle that volume also bring greater transparency in how FBI data is used, he said, insisting that privacy and data analysis aren’t a zero-sum balance. It’s possible to improve both, he said.

“If there is magic in [agile and continuous delivery]” Johnson said, returning to the stage, “it’s because we’ve enabled the entire workforce to bring their talents to the table … and collaborate in a meaningful timeframe,” Johnson said.

“We don’t think well about delivering something six years into the future. I haven’t been in a job for six years! Six years is a long time. Six months can be a long time. When the cycle times get that long, we fall into old, bad habits of spending our time writing documents in a silo,” he said.

Johnson doesn’t diminish the importance of planning, recalling former President Dwight Eisenhower’s famous line: “In preparing for battle I have always found that plans are useless, but planning is indispensable. ”

But continuous feedback and responsiveness ultimately changes the way organizations, not just IT departments, need to operate — by forcing “you to make a better decision today,” he said.