Reddy GeneratorsREDDY GENERATORSThe dependable power professionals

Control-System Redundancy, Failure and Degraded Modes

How to plan for control failure: failure modes of controllers, networks and power, defined degraded modes, manual fall-back, alarms for lost redundancy and tests that prove the plant stays safe.

In short. Every control system will eventually fail in some way. The design question is what happens next. Degraded modes define which functions continue, which stop and what operators must do, so that a failed controller or network never becomes an unplanned outage.

Failure modes

FailureTypical consequenceDesired degraded mode
PLC faultLoss of coordination functionsDevice controllers continue; manual sequences available
Network lossLoss of remote and PMS commandsLocal automatic operation
SCADA lossLoss of visibilityPlant continues; local panels used
Control power lossBreakers cannot operateBacked up supply. See control power
Sensor faultWrong decisionsVoting or plausibility checks; block rather than act
Software errorUnexpected actionsSafe state on watchdog; independent protection

Defining degraded modes

  1. List each control function and its dependencies.
  2. For each failure, decide which function remains.
  3. Define manual procedures that operators can follow.
  4. Define alarms that tell operators the plant is in a degraded mode.
  5. Define limits during degraded operation, such as maximum load or no paralleling.
  6. Define recovery steps back to normal.

Detecting lost redundancy

A redundant system running on one path looks normal. Monitor and alarm: single controller running, one network ring broken, one power supply down. See redundant control.

Manual operation

Provide local manual controls and written procedures for key actions: start, synchronise, transfer, shed. Train operators and drill. Keep keys, tools and procedures accessible.

Fail tests

See acceptance framework.

Illustrative example

A plant tests loss of its PMS during a utility outage. Generators continue to share load via their own controllers, the ATS continues to transfer, and operators receive a “PMS offline” alarm and a checklist that sets manual load limits. Loads are restored by hand, and the PMS rejoins safely when repaired.

Plan the failures

Every control system will have a failure at some point. The question is whether the plant degrades safely or collapses. Define degraded modes in advance.

FailureDegraded mode
SCADA offlineLocal HMI and controllers continue, operators use local panels
PLC failsRedundant PLC takes over; if both fail, generator controllers use stand-alone logic
Communication bus failsSets revert to droop sharing and local protection
Control power lossBreakers hold or fail to defined safe state
Sensor failureAlarm, use redundant sensor, fall back to conservative logic
Controller software faultWatchdog restart, hold last state

Design rules

  1. Each level can operate without the level above.
  2. Critical signals duplicated or voted two out of three.
  3. Watchdogs between controllers with defined actions.
  4. Manual operation always possible at the equipment.

Test the degradation

At FAT, deliberately disconnect each component and record the plant response. Repeat after major changes. A degraded mode that has never been tested is an assumption.

Operator procedures

Write short procedures for each degraded mode and train staff. Include contact details for support. See redundant PLC architecture and fail-safe architecture.

The master engineering question

For each control component, what happens to the plant in the first second, the first hour and the first day after it fails?

What to send us

Send the control architecture. Send your application details.

Questions engineers ask

Is full redundancy always needed?

No. Good degraded modes can be better than complex redundancy.

Who writes manual procedures?

Engineering with operations, then tested by drills.

How do I know the plant is degraded?

Through explicit alarms and a status display.

What about cyber incidents?

Treat as a failure mode with its own response plan.

This guide gives general principles. Confirm with a project design.

CallWhatsAppEnquire
STAGING PREVIEW ยท NOT INDEXED