The United States Postal Service lost the digital version of an entire database that records and monitors security incidents due to a failure of the hard drive that stored both the database and its backup, according to a report from the agency’s Inspector General.
Chuck McGann, CISO of the United States Postal ServiceUSPS Chief Information Security Officer Chuck McGann confirmed the loss of the digital Computer Incident Response Team (CIRT) database, but he said a hard copy record of each incident has been retained in the agency’s information security office in Raleigh, North Carolina.
“The requirement was for us to have this information,” McGann told FedScoop in a phone interview. “We had the information. We pointed them to the five file cabinets where we have physical copies of all of the incidents.”
USPS’s data management team backs up its critical databases off-site; however, the CIRT database was classified as an essential database rather than a critical one, according to the IG’s report, which said analysis of historical data and security record maintenance, like CIRT, is considered essential.
“We are issuing this alert to make the Postal Service aware of the need to modify its current backup and storage requirements to ensure that essential, but not critical, data is available,” the report said.
The platter, the magnetic circle in the middle of a hard drive, was responsible for the hardware failure at the United States Postal Service in April 2014 that resulted in the loss of the digital CIRT database. (iStock Photo)McGann said the specific failure came in the platter of the hard drive containing the backup and the production portions of the CIRT database – the platter is the circular portion of a hard drive where data is magnetically stored.
“Did the platter fail? Yes it did,” McGann said. “Did we in security know that the backup was on the same platter as the production, no we didn’t. Maybe we should’ve. I’m not saying yes or no; I’m not going to throw anybody under the bus.”
It is common practice, McGann said, for the production and backup portions to be stored on separate hard drives, but in this case, it was simply left unchecked.
“Operations 101 says that you don’t keep your backup on the same device that you keep your production,” McGann said. “I guess it was one of those things where, in hindsight, we should’ve checked.”
However, McGann did not know until the failure that backup and production were located on the same platter. In fact, around the time of the hardware failure, McGann and his team were working to migrate the Postal Service to a new system.
The migration was in the works for a while, McGann said, and the agency also discovered the hardware failure during the migration process. Now, in the new system, the agency has backups handled by a separate corporate entitity. In addition, USPS makes its own backup of the information as well.
In the end, due to the existence of the physical backup of the CIRT database, and due to the speed with which the agency addressed the hardware failure, the impact was minimal.
“Although the Postal Service took immediate corrective action for this database by implementing backup procedures on separate hardware, there may be other unidentified databases that are not backed up on separate hardware that could result in a loss of data and the inability to comply with record maintenance requirements,” the report said.
In response, the agency will conduct an audit of its other systems to determine whether or not a similar problem could happen in another database or another system elsewhere in the agency.
The initial hardware failure was discovered during the fiscal year 2014 information technology internal controls audit conducted by Kimberly Benoit, deputy inspector general for information technology and data analysis.
In its final recommendations, the IG recommended that McGann prohibit backups from being stored on the same hardware as the entity being backed up. These changes will be implemented by April 2015, the report said.
