Ethics in Software Engineering

SENG 365 — Software Engineering

Neil Ernst
nernst@uvic.ca

University of Victoria

2026-10-01

Before Class

Read Section 1: General Ethical Principles of the ACM Code of Ethics.

It is seven short principles. Bring one principle that seems easy to agree with but difficult to apply.

Learning Outcomes

By the end of class, you should be able to:

  • identify who may benefit or be harmed by a software decision;
  • connect a plausible consequence to a professional obligation;
  • distinguish accountability from blame; and
  • turn an ethical concern into a requirement, test, disclosure, or stopping rule.

Outline

  1. Three software cases -> frame the challenge
  2. 1-2-4-All -> Black mirror episode pitch
  3. What, So What, Now What -> Responsibility map and redesign
  4. Individual exit note -> One required disclosure

Professional Responsibility

Code Is Not the Boundary of Responsibility

Something can be:

  • technically correct but harmful;
  • legal but unfair;
  • requested by a client but contrary to the public good; or
  • tested but still unsafe outside the test assumptions.

Ethical judgment moves us beyond “the ticket said so”

A Vocabulary for the Conversation

From the ACM Code:

  • 1.1 Contribute to society and human well-being.
  • 1.2 Avoid harm.
  • 1.3 Be honest and trustworthy.
  • 1.4 Be fair; do not discriminate.
  • 1.6 Respect privacy.
  • 2.5 Evaluate systems and risks thoroughly.
  • 3.1 Make the public good the central concern.

Three Software Cases

Therac-25: Safety Delegated to Software

Between 1985 and 1987, six known accidents involving the Therac-25 radiation therapy machine caused massive overdoses, deaths, and serious injuries.

  • Software carried nearly all responsibility for safety, yet the initial safety analysis did not include it.
  • Timing-dependent software faults could place the machine in an unsafe state.
  • Independent hardware shutdown and interlock mechanisms were added later.

Therac

What evidence should be required before software replaces an independent safety barrier?

Relevant principles: avoid harm (1.2) and evaluate risks thoroughly (2.5).

Tesla Autopilot: A System-Level Failure

In a fatal 2018 Mountain View crash, a Tesla Model X operating with partial driving automation steered into a highway nose area.

The regulator identified interacting causes (swiss cheese model):

  • limitations in the Autopilot system;
  • driver distraction and overreliance;
  • ineffective monitoring of driver engagement; and
  • a damaged, unrepaired crash attenuator.

Tesla

When foreseeable human overreliance interacts with software limitations, what must the design prevent, detect, or communicate?

Source: US National Transportation Safety Board, Mountain View crash investigation.

Knight Capital: Deployment Is Part of the System

In 2012, incorrectly deployed code activated a defective, dormant function in Knight Capital’s trading router.

  • The system sent millions of orders while processing 212 customer orders.
  • Ninety-seven error emails arrived before the market opened but were not treated as actionable alerts.
  • In 45 minutes, the firm accumulated unwanted positions and lost over US$460 million.

Knight

When does a log message, test, or review become credible evidence rather than a box that was checked?

Source: US Securities and Exchange Commission, Knight Capital enforcement release.

The Recurring Pattern

Across all three cases:

  1. software interacted with people, hardware, procedures, and incentives (socio-technical);
  2. warning evidence existed but did not reliably change a decision;
  3. responsibility crossed organizational and technical boundaries; and
  4. the cost of recovery was borne by people with unequal power.

Keep these questions for the fictional future:

What evidence was available? Who could act? What made inaction seem reasonable?

Opening Decision

An AI coding agent implements a feature. It passes the tests, a human approves the merge request, and the feature later causes harm.

Who is accountable, and what would accountability require before release?

Write one sentence. Avoid using “the AI” as the only responsible actor.

Black Mirror Writers’ Room

The Structuring Invitation

Three years from now, what plausible system could begin with a genuine benefit but shift power or cause harm because of an ordinary software-engineering decision?

Choose one current technology seed:

  1. coding agents that implement and review features;
  2. biometric or behavioural inference;
  3. automated eligibility, risk, or fraud decisions; or
  4. reputation scores that move between platforms.

Push it one plausible step, about three years into the future.

1-2-4-All: Create the Episode

1 — Alone (2 minute): Write a protagonist, a benefit, and one possible turn.

2 — Pairs (4 minutes): Share both ideas. Build on them; do not evaluate yet.

4 — Quartets (8 minutes): Select or combine an idea. Complete the pitch:

  • the promise and protagonist;
  • the ordinary design decision that creates the turn; and
  • who gains power, who loses options, and what becomes hard to undo.

All (5 minutes): A few quartets share one turn that everyone should hear.

What, So What, Now What

Stay Low on the Ladder

Move through the questions in order:

  1. What? Facts and observations. Avoid trying to explain.
  2. So What? Meaning, consequences, power, and obligations.
  3. Now What? Actions and experiments that follow from the analysis.

Do not solve the problem while the group is still establishing what happened.

What? Trace the BM Decision

5 minutes: 1 alone → 3 in your quartet → 1 capture.

At the moment when another future was still possible:

Actor What did they know? What could they control? What did they owe?
Developer / reviewer  
Product or organizational lead
Operator or customer
Person affected

So What? Apply the Code

6 minutes: 1 alone → 4 in your quartet → 1 capture.

Choose two ACM principles and ask:

  • Which principle creates the clearest obligation?
  • Who benefits, who is harmed, and who has the least power?
  • What observable evidence would show that the obligation was met?
  • Who had the authority to request or act on that evidence?

Reflect

Now What - Exit Note

Complete both sentences:

When AI assists engineering work, the team should be required to disclose __________ because __________.

One piece of evidence that must remain attributable to a human is __________.

Submit your note before leaving.

Sources

Activity and Reading

Software Cases