@readmodelmag
MODEL

Tech

Workday's Friday outage started with a backup test. Here's what is confirmed

SummaryWorkday, the HR and payroll platform used by more than 11,500 organizations, put some customer systems into maintenance mode on Friday, Nov. 13, after a problem found while testing its backup systems left some data in an inconsistent state.

· MODEL Editors · 2 min read

Some links may be affiliate links. Disclosure

Cartoon meme titled Testing disaster recovery on Friday the 13th. A grinning server labeled BACKUP wears a party hat and holds a gold trophy, with its orange cable plugged into a dizzy server labeled PRODUCTION behind a striped barrier and traffic cone. A tired office worker with coffee stares at a laptop error screen, under a wall calendar with a black cat. Caption: Good news: the disaster part works. Watermark: MODEL · readmodelmag.com.
Testing disaster recovery on Friday the 13th. Illustration by MODEL. Original. Image source.

What happened. Workday, the Pleasanton, California, company whose software handles pay information, time-off requests and other HR tasks, had a widespread outage on Friday, Nov. 13. The company said it placed some of its systems into maintenance mode, leaving some employees at affected companies unable to log in or finish tasks (San Francisco Chronicle (opens in a new tab)).

The cause, so far. Workday said the disruption began after a problem was discovered while testing its backup systems, and that the issue left some information in an inconsistent state (San Francisco Chronicle (opens in a new tab)). A customer notice posted by Florida Atlantic University's IT office at 8:00 p.m. ET said Workday had found an issue involving its Disaster Recovery environment that resulted in inconsistent data in production, and that Workday was placing all WD1 production tenants into maintenance as a precaution (FAU OIT (opens in a new tab)).

What Workday said. "Some of our customers are experiencing a service disruption and we're working urgently to investigate and resolve the issue. We sincerely apologize for the challenges this is causing impacted customers," a Workday spokesperson said (KRON4 (opens in a new tab)). Workday said it would give updates every 30 minutes (San Francisco Chronicle (opens in a new tab)).

How big. Workday is used by more than 11,500 organizations (San Francisco Chronicle (opens in a new tab)). Downdetector had more than 2,900 reports of Workday problems as of 3:15 p.m. PT, and some users saw "Workday is currently unavailable" at login (KRON4 (opens in a new tab)). Some employers use Workday for benefits open enrollment, which employees could not complete during the outage (LiveNOW from FOX (opens in a new tab)). Emerson College, for example, said it would keep open enrollment open until noon ET on Monday, Nov. 16 (Emerson College (opens in a new tab)).

What we don't know. As of the reports above, Workday had not given an estimated restoration time, and it has not said how many customers were affected. Lists of big-name Workday customers in some coverage are lists of customers, not confirmed outage victims.

The builder lesson. Disaster-recovery drills are supposed to prove the standby can take over without touching live data. If you run your own DR tests: make sure standby services cannot reach production databases (separate credentials and network rules, not just config flags), and don't schedule a risky drill at a time when your users can't afford downtime, like a payday Friday afternoon.

This story will not be updated live. Check Workday or your employer's IT status page for current service status.

Get three tested tools every Friday

Free, weekly. Unsubscribe anytime.

Sources

  1. Workday outage leaves clients unable to access accounts

    San Francisco Chronicle

  2. Workday users experiencing 'service interruption'

    KRON4

  3. Workday outage means employees across thousands of businesses can't access accounts

    LiveNOW from FOX

  4. Workday Production Service Disruption

    Florida Atlantic University OIT Status

  5. Workday Is Currently Unavailable

    Emerson College System Status

FREE · EVERY WEEK

The top AI tools this week, tested.

Every week MODEL's editors pick the AI tools worth your time, try them hands-on, and tell you what's good, what's not, and who it's for. Free, in your inbox.

Free. Unsubscribe anytime. Our picks are editorial and can't be bought.