Dan Morgan takes a deep dive into DOT’s open data progress

The Department of Transportation has been working since late 2016 to build out a centralized portal for its data — and it has no shortage of data.

States send the department data on the condition of their roads and how many cars are traveling on them annually, for example. The department has data on rail-related injuries and deaths, data related to weather, data on worker fatalities on the railroad — the list goes on and on.

It’s a lot of data to maintain and make available to the public, and the department was focused on creating “one good customer experience” for its vast amounts of information on everything from road conditions to annual traffic, said its Chief Data Officer and acting Chief Technology Officer Daniel Morgan.

Morgan showcased the progress the department has made so far building out the repository, and the lessons it’s learned along the way, at Socrata Connect in Austin last week.

“Our goal was to have no wrong door, and one good customer experience for all of that data, and to make it easy for our data publishers to get data out to the public,” he said. “I think the ultimate goal here has been to build a truly seamless experience around our data.” To do that, the department borrowed the Digital Services Playbook from the U.S. Digital Service, honing in on its goals of building with an agile, user-centered approach.

DOT also developed the portal through a shared services model in which component agencies contributed funds to data.transportation.gov, Morgan said.

“The shared services approach we took was to seek out ‘early adopters’ to help share the cost of the service and then to onboard them as they became ready to take advantage of the service,” he told FedScoop after his presentation.

DOT built data.transportation.gov using the Socrata platform, starting with its administrative data sets, many of which come from state and local partners.

“But the federal role isn’t just in administering these big administrative data sets that are useful for policymaking, economic analysis, longitudinal trends,” he said. “We’re also researchers.”

One of DOT’s clients for the shared service is the Intelligent Transportation Systems Joint Program Office, which has a research mission to provide “timely access” to public data from connected vehicle deployments around the country. The timely part is not so easy.

“A lot of research data tends to come out at the end of a project, but these things go on for a real long time, and then it takes a long time to get the research data put together, and put a paper together, and run it through a journal,” Morgan said. “And that’s going to slow down your ability to discover what that data looks like, and be able to do third-party research, evaluation and acquisition development.”

So the department decided it needed one solution to handle massive amounts of data coming from connected vehicle pilots in Wyoming, Tampa, Fla., and New York City.

“It needs to be big — we’re talking about terabytes of data,” Morgan said.

They started to discuss the issue last August and launched the ITS Public Data Hub in January as an easy entry point for data on intelligent transportation systems.

The goal was to “break Socrata,” Morgan joked.

“We kind of did, but that’s okay because we were able to handle it,” he said.

Data from the Wyoming connected-vehicle pilot is available now, and data from the Tampa and NYC pilots are expected to be made available in mid-2018.

“We have to share. And we can’t wait three years in a technology field that is moving so quickly, in an industry that is changing so rapidly, for results to finally come up from research,” Morgan said. “So we have to push the boundaries to get knowledge to be shared more quickly.”

Asked for advice based on his experience with the portal and ITS Public Data Hub, Morgan said to pick something that’s manageable and learn from it.

“You have to learn how the process needs to work. So being able to run something all the way through and gather your lessons is fundamental to really being able to do it at scale,” he said. “So don’t be afraid to try, but pick your pilot the right way so that you can complete something.”

What’s next for Morgan? Replicating the success of the ITS Public Data Hub “as many times over as we possibly can.”

“I think we’re going to continue to invest in data.transportation.gov for more use cases,” he said. “This ITS Public Data Hub was a great collaboration between a research area and the chief information officer, and me as the CDO, working together to sort of prove out a concept.”