Every engineering organization has at least one person it has quietly rearranged itself around.
Not a monster. Just the colleague who makes every interaction cost a little more than it should. The one whose pull requests turn into sieges of nitpicks that have nothing to do with whether the code is correct, who reopens the decision from three weeks ago over a fourth edge case that will never occur, who answers a simple question with a lecture and corrects the grammar in your incident writeup while the incident is still burning, and who is, above all, unpleasant in the small continuous way that never quite clears the bar for a real conversation but leaves the room a little more tired than it found it. You know the one. You are already picturing a specific face.
The industry has a favorite story about this person, and it tells the story as a compliment. He is brilliant, the story goes. So far above the median, so load-bearing to the roadmap, that the damage is just the weather you accept in exchange for the genius. You do not fire the weather. You buy an umbrella, you route the standup around the storm front, and you tell the new hire in a low voice that yes, he can be a lot, but he is worth it, as though you were handing over a piece of institutional wisdom rather than a warning.
It is a strange thing to believe, and stranger the moment you notice how seldom it is even true. Most of the colleagues who cost the most are not brilliant. They are ordinary engineers having an ordinary career, and the brilliance, where it exists at all, is usually the alibi reached for after the behavior was already tolerated, and not the reason it was tolerated at all. The story is not describing a genius. It is describing a decision, dressed as a compliment so that nobody has to defend it out loud.
It is stranger still because we do not believe anything remotely like it about machines, and we have spent enormous intellectual effort making sure we don't.
The entire discipline of distributed systems is, if you squint, a decades-long argument about how things are allowed to fail. We assume the disk will die, the process will get OOM-killed, the network will partition at the worst possible instant, the region will go dark during the launch. This is not pessimism, it is the operating condition, and what separates a senior engineer from a junior one is often just the number of failure modes they have learned to take personally.
But here is the thing the org chart never internalized, even though every engineer on it knows it in their spine.
There are two kinds of failure, and they are not close.
The Honest Failure
The first kind is the machine that crashes.
A crash is, in a real sense, a polite failure. The node stops. The heartbeat goes missing, the health check times out, and that silence is itself a clean signal: the system knows something is wrong, sheds the dead replica, and keeps the quorum. We call such a component fail-stop, because it dies honestly or not at all, and never does the one truly dangerous thing.
The dangerous thing is to keep answering.
The second kind of failure is the node that does not stop. It stays up, stays in the quorum, and keeps responding, on time, with answers that are wrong, or late, or beside the point, or technically valid and useless, or just a flood of objections to things already settled. It does all of this while looking, from the outside, exactly like a healthy participant, which is the entire problem: you cannot fail over a node that has not failed in any way your monitoring can see.
In 1982 Leslie Lamport, Robert Shostak, and Marshall Pease named this failure, and the name stuck for forty years because the problem never went away. They called it the Byzantine Generals Problem, after a fable about generals who cannot coordinate a siege because some of them behave unreliably. The fable is usually told about traitors, but the result outgrew it. What made the problem hard was never the malice; it was that the bad node keeps participating. A node that fails by any means other than cleanly stopping, whether it lies or is merely erratic, obstructive, or exhausting, is Byzantine, and the verdict is arithmetic. To tolerate a crashed component you need a bare majority, roughly 2f+1 nodes to survive f failures. To tolerate a Byzantine one, alive and misbehaving, you need 3f+1: four honest participants to safely absorb a single unreliable one.
Read that ratio again, because it is the whole essay in one number.
The node that dies costs you a spare. The node that stays up and misbehaves costs you most of a fault-tolerant system built to work around it.
Categorically more expensive, by design, in the math. And notice that nowhere in that proof does anyone have to be a villain.
Difficult Is a Crash. Toxic Is Byzantine.
Now hold that distinction next to the people, because it is the same distinction, and the industry has spent a fortune failing to notice.
There is an engineer, and every good team has had one, who is difficult the way a cold shower is difficult. He tells you in review that your abstraction is wrong and does not soften it, and it stings, and he is usually right, and when he is wrong you can say so and he updates, because what he cares about is the system being correct, not you being small. He is loud, impatient, a genuine pain on a bad Friday, and completely legible. His signal is trustworthy even when his tone is terrible, because he tells everyone the same thing to their face.
That person is a crash fault. Fail-stop, honest. You route around him occasionally, it costs a spare, it is worth it, and the umbrella metaphor more or less holds.
He is not who the ghost story is about.
The person the ghost story is about is different, and not because he is worse in some dramatic way. It is that his failure keeps running. Those behaviors from a moment ago, the endless review, the reopened decision, the lecture, the small steady rudeness, share one property, and it is the one that matters: he does not lie, he does not need to, he just never stops emitting, and every unit of it is something the rest of the team is now obligated to process.
That is Byzantine in the only sense that matters here. Not treachery, just continued participation. He has not crashed, so you cannot fail over; he is still in the quorum, still reviewing, still technically doing the job, and the cost is never any one thing he says. It is that his presence taxes every transaction that routes through him.
Robert Sutton, who has spent a career studying this, reduces the diagnosis to one test that survives all the noise about intent: after dealing with the person, does the other party walk away feeling worse, smaller, more drained, less sure of themselves than an hour before? That test is precise. It describes a node that does not merely underperform but writes its degradation into everyone it touches. You leave a little less certain of work you were fine about yesterday, and you will serve that corrupted value back to yourself for a week. The fault propagated, and that is what makes it expensive, entirely independent of whether he meant anything by it.
The difficult engineer fails honestly, and you can build around him.
The toxic one keeps participating, and every interaction costs the rest of you something you never get to bill him for.
The Fault Propagates on Read
The reason a Byzantine fault is expensive is not that the bad node is bad. It is that the bad node is connected.
A corrupted value does not stay politely inside the machine that produced it. In a gossip protocol each node periodically syncs state with a few neighbors, they sync with theirs, and a single bad entry rides the mesh outward until a real fraction of the cluster is confidently serving the same wrong answer and citing each other for it. The downstream nodes did nothing wrong. They read from a peer they had no reason to distrust.
Now stop pretending this is about databases.
The research on toxic teams reads almost like a paper on epidemic broadcast. Will Felps ran the study everyone suspects is true and would rather not measure: drop one deliberate bad actor into a capable small team, one person withholding and belittling and undermining, and team performance falls by thirty to forty percent. Not his performance. The team's. The honest nodes degrade because a large share of their compute now goes to reading the room, pre-drafting the defensive reply, and verifying whether the last interaction meant what it seemed to.
Two more findings stack on top. A negative interaction lands with roughly five times the force of a positive one, so the quiet ledger where his output nets against his friction is off by an order of magnitude before it starts. And the mechanism sits under a famous small experiment: give someone a trivial, arbitrary bit of power over two others, put a plate of cookies on the table, and watch them take the extra one and chew with their mouth open. Power teaches by demonstration. The team learns what is tolerated and updates its own protocol, and now the corruption is not in one node but in the culture's shared state, the layer nobody knows how to roll back.
That is contagion, the fault propagating on read, and it is why the isolated-genius accounting fails. You cannot evaluate a Byzantine node in isolation, because isolation is the one thing it never is.
The Wrong Ledger
So watch what an organization does when it meets one of these people, because the tragedy is in the competence of the mistake. It runs a cost-benefit. Value on one side, the revenue he touches, the systems he knows, the fires he puts out; cost on the other, the complaints, the attrition, the HR hours, the manager's Sundays. It does the subtraction, the subtraction comes out positive, he stays, and everyone congratulates themselves on their maturity before a hard tradeoff.
That subtraction is the crash model. Value minus friction, net contribution, the clean accounting you would apply to a component whose only sin is a little downtime.
But a participating failure does not subtract. It taxes. The cost is not the friction he emits in any one exchange, which is at least countable. It is the toll everyone else pays to route work through him: the review that now takes three days and a private side-conversation to survive, the meeting people over-prepare because one comment will derail it, the good idea not raised in his channel because it is not worth the six paragraphs of objection, the junior who stops asking questions. The 3f+1 shows up here as real, continuous spend, the whole apparatus of working around and bracing for and handling-with-care, all of it filed on the books as the ordinary overhead of management. It is not ordinary. It is the redundancy tax on a participating fault, paid in full and named something else so it never reaches the ledger where the decision gets made.
Which is the first honest thing to say. The brilliant jerk survives the cost-benefit not because he is worth it, but because the org is pricing him with an instrument built for a different failure, one that physically cannot see the expensive part.
Why You Cannot Just Route Around Him
It would be satisfying to stop there with a tidy imperative: find the Byzantine node, pay the small cost of removal, restore the quorum, go home. Fire the assholes. The internet loves that ending and it is not entirely wrong. It is just too easy, and it skips the real question, which is not what should happen in principle but what to do on Monday, when he is still here and the org has already run its arithmetic and decided he stays.
The obvious move, routing around him the way failover routes around a dead replica, so often fails for a reason that is uncomfortable but not the one the angry reader of this essay wants. It is not that leadership is corrupt. It is that the layer meant to settle the dispute is running the same broken accounting one level up, in good faith. A manager states the respect value in the all-hands and means it, then, when honoring it this quarter would mean losing the person who touches the revenue, decides this is not the hill and you should try to work with him. He is not being two-faced. He is optimizing locally, the way the fastest engineer ends up handed every hard ticket until the knowledge lives in one skull, each call defensible, the sum quietly catastrophic. He is doing subtraction where the situation calls for fault tolerance, and he is doing it sincerely, which is worse in its effects and far better in its prospects, because you cannot argue a person out of malice but you can hand him a better model. The rest of this is an attempt to hand out two, one for each side of the org chart.
Navigating Him as a Peer
Start from the humbling premise that you are not going to debug him.
You are an engineer, so your reflex when something behaves wrongly is to reproduce the fault, isolate it, and fix it. He is not a defect in a system you own. The attempt to fix him is, in fact, the exact mechanism by which you get pulled into the mesh and start spending your own cycles tracking his state, which is the failure mode dressed up as diligence. So drop it. The goal is not repair. The goal is to stay legible, intact, and useful while the org takes however long it is going to take.
There is a distributed-systems reflex here worth naming just so we can set it down. The usual way you tame a badly behaved node is to force everything onto a shared, visible channel where it cannot get away with anything in a corner. In a remote, everything-logged company, that fight is mostly already won for you. Slack keeps the receipts. The pull request is a permanent public record. The decision is in the doc because there was nowhere else to put it. Documentation is not your problem, and if it were, the fully remote team would be the healthiest place on earth, which anyone who has worked on one can tell you it is not.
So the lever is not where the record lives. It is where your attention goes, and where his output is allowed to land. He is not hiding anything and he is not lying, which means there is nothing to catch and no case to build. He is simply generating more traffic than the work requires, and counting, without ever once knowing that he is counting on it, on you to process every byte. The whole move, on the peer side, is to stop being the node that processes every byte.
Robert Sutton wrote an entire field guide for the target's side of this, and the striking thing when you read it as an engineer is how many of his tactics are already sitting in your standard library under other names.
The first one he keeps returning to is that the behavior runs on your reaction, so you starve it by controlling the timing of your response. He calls it slowing the rhythm; you already call it backoff. The nitpicker and the re-opener and the lecturer are running an interrupt-driven loop whose interrupt is your engaged, slightly wounded reply, and every fast reply is the ACK that keeps the loop hot. So stop ACKing on his clock. Answer the twelve-comment review in one pass at the end of the day rather than twelve times in real time. Let the thread cool before you respond. Reply to the six-paragraph objection with two sentences. You are not sulking. You are declining to let a peer's volume set your pace, for the same reason your services back off a host that will not stop retrying.
The second tactic Sutton spends a chapter on is avoidance, which sounds cowardly until you translate it into coupling. You reduce your coupling to the node. Remote work makes this nearly free and almost nobody spends it: you control your own fan-in, so drop off the thread where your presence is not necessary, skip the meeting that produces a written summary, convert a standing sync to async, and route your real questions to anyone else who can answer them. On-site, buying distance from a difficult colleague meant physically hiding in a stairwell. On a distributed team it is a mute on a channel that no one else will even notice, and every edge you delete is one less path along which he can tax you.
The third tactic is the one that matters most and is the hardest to run, because it executes inside you. Do not persist his read of you to durable storage. Sutton gives this a third of the book and it is cognitive behavioral therapy wearing a hard hat: the lasting damage a low-grade toxic peer does is not to your calendar but to your self-estimate, the way you quietly start serving yourself a worse number for your own competence a week after he implied one. Treat his verdict as untrusted input and validate it before you write it anywhere permanent. He said the framing was naive; was it, when you checked it against someone whose judgment you actually trust? Keep his blunter reframes in a pocket, because they are load-bearing: it is not your fault that he is like this, this will feel small by bedtime, and the flat minimum-viable acknowledgment ("noted, thanks") that hands the loop nothing to feed on.
The fourth is the one people skip because it feels like an admission of weakness, and it is the single most evidence-backed move in the entire literature. You cannot establish ground truth about a Byzantine node alone; the whole theory says so, it takes a quorum. So assemble one. Compare notes with the people who work near him, not as a whisper campaign but because the first thing corroboration returns is the discovery that you are not inventing it, which is precisely the value he corrupted. Sutton has the numbers and they are not subtle: when targets banded together to push back, the offender was sanctioned about 58 percent of the time and none of them were fired, and when people went it alone the sanction rate fell to 27 percent while one in five of the isolated lost their own jobs. Solidarity is not a feeling here. It is the difference between a signal the system acts on and one it drops.
That same quorum is what dissolves the review that will not end. Do not out-argue the merge held hostage to a dozen nitpicks, because arguing is the reaction and the reaction is the fuel. Route it to the standard instead of to him: pull in a second reviewer and ask, without heat, for each blocking objection stated against the team's actual bar. A surprising amount of pedantry cannot survive being asked to justify itself as a merge-blocker to a room, having only ever existed as a comment aimed at a person, and the objections that turn out to be merely optional the team can now decline together, which is the one thing no individual ever feels safe doing alone.
None of this fixes him, and none of it is trying to. Backoff, reduced coupling, input validation, and quorum: it is exactly the kit you would reach for against any component you cannot trust and cannot yet remove. It keeps you whole and legible and hard to destabilize while the slower machinery upstairs takes however long it is going to take.
Navigating Him as a Leader
If you are the one who can actually allocate, the good news and the bad news are the same sentence. This is yours to fix and no one else's, because you are the only node with write access to the shared state.
Begin by throwing out the subtraction. The instinct to weigh his output against his friction is the exact error, because you are pricing a participating fault with a crash-fault instrument and the expensive part is invisible to it. So go instrument the expensive part, which Sutton calls the total cost and which is not actually hidden, only unglamorous. Who has quietly asked to move off his projects. Where the regretted attrition clusters. How many of your hours, and your skip-levels' hours, go to managing around one person. Whose good ideas stopped showing up in the rooms he is in. In a remote org those signals arrive scattered and mostly aimed at you, one resignation call and one skip-level aside at a time, which means you are often the only node positioned to assemble them into a pattern, and therefore the only one who can. That is the redundancy tax, and the moment you put it on the same page as his output, the arithmetic usually inverts, not because you decided to be tougher but because you finally measured the right quantity.
Most of the work, though, happens long before anyone is removed, and here the two books say the same thing in different dialects. Sutton's largest real-world result did not come from a values poster; it came from a program that had people correct the small stuff in the moment, the glare, the interruption, the person talked over, and measurably drove the behavior down. Larson says it from the management side in two lines worth taping to a monitor: feedback for weak performance should be delivered immediately, and surprise is the cardinal sin of performance management. The node you correct on a quiet Tuesday, specifically and without ceremony, is the one you never have to evict in Q3. What you cannot do is let the small emissions run for a year and then act shocked at the total.
And some of what you are about to diagnose as toxic was manufactured upstream, which is Larson's sharpest contribution to this. A team pinned at a hundred percent utilization produces irritability, corner-cutting, and short fuses as a matter of physics rather than character, the way a saturated link starts dropping packets that were perfectly good. If the whole team has gone a little Byzantine at once, you do not have twelve bad apples, you have a slack problem, and the remedy is capacity and sequencing, not a talking-to. Protect the slack and a surprising amount of the ambient nastiness resolves itself without anyone being named.
When correction genuinely fails, though, evict, and evict sooner than feels comfortable, because Sutton has both the pattern and the receipts. Men's Wearhouse fired a top salesman who would not stop poisoning his store, and the store's sales rose about thirty percent; more universally, he found that the reaction to finally removing one of these people is almost never regret, it is a stunned why did we wait so long. The thing that stalls the eviction is the same accounting error from the ledger, the belief that his throughput makes him too costly to lose, so overwrite the belief with the correct one. A node that returns corrosive or wrong output is not a fast node, it is a broken one, and "brilliant but toxic" should read to you exactly the way "fast but wrong" reads in a benchmark. Treat the certified case as an incompetence problem, because at the level of the team's actual output, that is what it is.
Then, and most importantly, be a consistent node yourself. The mixed signal is not corruption, but it is still the most damaging thing you can emit, because your inconsistency is what tells the whole team that the stated values are decorative.
Say the same thing in the one to one that you say on the wall. A value has to survive contact with a bad quarter at least once, visibly, or it is not a value, it is a poster, and everyone can already tell the difference. You do not have to be dramatic about this. You have to be legible. The entire purpose of an adjudication layer is that people can bring it a problem and get a consistent answer, and the moment they can, a startling amount of what people call toxicity simply drains out of the system, because it was never only the one node. It was the uncertainty about whether anyone with power would ever agree that the node was real.
Two cautions, because the cleanup has failure modes of its own. Do not, in a fresh burst of resolve, purge the difficult-honest engineer along with the toxic one. He is a crash fault, loud and legible and trustworthy, frequently your best early-warning system precisely because he will say the unwelcome true thing to your face. If your intervention quietly teaches the team that bluntness and cruelty are the same offense, you have not raised the bar, you have merely optimized for pleasantness, and pleasant and honest are different variables that occasionally correlate. And do not mistake the removal for the whole job. The node is the acute problem; the accounting is the chronic one. Remove him without changing the math that selected for him, and you will grow another, because the system, as the old line goes, is perfectly designed to produce the results it produces.
What the 1980s Already Paid For
None of this requires anyone to be a villain, and that is the reading I most want to survive the text, because the cheap version of this essay is a morality play and the true version is an accounting error that most of the building is making together, in good faith, with the best instrument they were ever handed.
We paid for the better instrument twice. The first time was 1982, in Lamport's proofs, which turned the cost of tolerating a Byzantine node into a theorem rather than an opinion. The second time was the 2000s, when large systems started hitting the fault for real, disks and servers that did not crash but silently handed back corrupted data, and the industry had to build Byzantine fault tolerance into production to survive it. Both times the verdict was identical. Lamport had been right, and expensive, all along. We know, with the certainty of mathematics, that the failure which keeps participating is worse than the failure which stops. We architect our most trusted systems, the ones that move money and hold consensus, on exactly this insight, and we would call an engineer negligent for designing anything critical as though every node still returning a response must therefore be telling the truth.
And then we walk into a room full of people and run the fail-stop math anyway, and call the result culture.
A machine that dies tells you the truth. It stops, and its silence is a gift, and you build the small honest redundancy that survives it. The one that stays up and keeps answering, wrong or slow or endless or just tiring, is the one the whole theory exists to survive.
We have known which failure is more expensive for forty years.
The only part left is to spend the knowledge on each other.