While most IT workers were told it’s a temporary detail, several say they are skeptical they will ever return to their tech-centered jobs.

The shakeup follows warnings from the Treasury’s watchdog about how staffing cuts at the IRS could affect tax filing season.