Previous Article:

We have just built our small kingdom. One valley, a few villages, a few signal towers, and a local network that works surprisingly well for something invented by people who still believe that medicine is mostly a matter of balancing four bodily fluids.

Unfortunately, sooner or later power becomes the main goal of the most ambitious people in the kingdom, and sooner or later one kingdom becomes too small for them. Scouts discover settlements behind the mountains, armies march along roads that were previously used for sheep, and within a few years our neat little kingdom becomes an empire with several valleys somewhere beyond the horizon.

The moral dilemma is outside the scope of this article, and we, as network engineers in medieval Europe, are definitely not in a position to judge the military policy of our lords. What we do need to judge is whether the communication system from Part 1 can survive this expansion.

It cannot.

Our old system knows how to move a frame inside one valley, but it has no idea what a valley is, where its border lies, or what to do when a destination is hidden behind three mountain passes and somebody else’s tax system.

So we will build the larger system in the same way we built the local one. We will remain with our towers until messages can cross the whole empire, and only after the machinery works will we return to a real computer and the Internet. This order matters because IP addresses, gateways and routing tables are much easier to remember when they arrive as answers to problems we have already felt.

In this post we will discuss how IP addresses, network prefixes, gateways, routing tables and ARP turn local networks into a network of networks. We will then see how the same model appears on Linux, how DHCP gives a host the information it needs, and what changes while one packet crosses several routers.

The real-world part focuses on IPv4 carried over Ethernet. Other link technologies can carry IP without Ethernet, and IPv6 uses Neighbor Discovery instead of ARP, but one carefully bounded world is enough for now. Our empire is already difficult to govern without opening several additional continents.

Subscribe now

Two orders of names

In our existing system every tower already has a local name such as 0A, 0B or 0C, and that worked perfectly while the entire known world fitted inside one valley.

Then we acquired another valley.

It has its own towers, which may be called 1A, 1B and 1C. They may also be called 0A, 0B and 0C, because coordination between medieval kingdoms is apparently not much better than coordination between modern IT departments.

Now an order addressed to 0A is ambiguous. There is an 0A in Valley 1 and another 0A in Valley 2, and the old name tells us who should receive a message only after we already know which local network we are talking about.

This is not a flaw in the old system. A local name was invented for local delivery, and inside one valley it still does that job perfectly. The empire has simply created a second question that the local name was never meant to answer:

In which valley does the destination live?

So we introduce another order of names. Tower 0A keeps its local address and independently receives the wider address V1.7. A tower in Valley 2 may reuse the same local address 0A while independently receiving V2.12. The local and wider addresses are not constructed from one another. They are separate identifiers assigned for different scopes.

Figure 1. The local address 0A can be reused in both valleys. The independent wider addresses V1.7 and V2.12 identify the towers across the empire, and neither is derived from 0A.

We have just reconstructed the reason a machine can need both a MAC address and an IP address.

The MAC address is the local name. It identifies the immediate sender or receiver on one Ethernet network. The IP address is the wider name. It identifies the source and destination across many connected networks.

The two addresses do not compete, and one is not a more modern replacement for the other. They describe the same machine at different scopes, rather like a person’s room number and full postal address. A room number can be perfectly useful inside one building even though a letter crossing the country needs more information.

Is the destination still inside our valley?

Imagine that tower 0A in Valley 1 receives an order:

Deliver this message to V1.12.

The destination begins with V1, and our tower also belongs to V1, so the destination is local. It can use exactly the machinery we built in Part 1. It discovers which local tower owns V1.12, learns that tower’s local address if necessary, and sends a frame through the valley.

Then another order arrives:

Deliver this message to V2.12.

This time the wider address tells us something important before a single signal is sent. The destination belongs to Valley 2, while our tower belongs to Valley 1. None of the local towers can deliver a frame directly through the mountain.

Our medieval rule can still be very simple:

V1.*  local
everything else  beyond the valley

Modern networking calls the precise version of this boundary a network prefix, but we do not need the binary notation yet. For the moment, the important discovery is that an address must contain enough structure to tell us not only who the destination is, but also whether it belongs to our local world.

Once 0A knows that V2.12 is remote, it needs somewhere local to send the message.

A gateway at the edge of the valley

We build a special tower at the border of Valley 1. It can receive local frames from our valley, but it also has a road leading into another network. Let us call it G1.

Tower 0A now learns one additional rule:

If the final destination is outside Valley 1, give the packet to G1.

This creates a situation that looks strange only until the two scopes become visible. Suppose 0A wants to reach V2.12. The final destination is still V2.12, but the next tower that can physically receive a local frame is G1.

The message therefore carries two destinations:

Final IP destination: V2.12
Local MAC destination: G1

Figure 2. The local frame is an outer delivery envelope addressed to the next hop, G1. The packet inside keeps the final destination, V2.12.

The outer address gets the message across the current local network. The inner address survives beyond that network and tells every gateway where the journey should eventually end.

Gateway G1 receives the frame because its own local address is written on the outside. It removes that local envelope and reads the wider address inside:

Destination IP: V2.12

Now G1 faces the same question that tower 0A faced, only on a larger map. Where should this packet go next?

A small map inside every gateway

We hang a parchment inside G1:

Valley 1          directly connected
Valley 2          eastern relay R1
Valley 3          northern relay R2
Everything else  royal highway G0

This parchment is our routing table.

It does not describe the complete road from Valley 1 to every tower in the empire. G1 does not need to know every bridge, every mountain pass or every official who will lose the paperwork. It only needs to know the next useful step.

For V2.12, the table selects relay R1. If the link between G1 and R1 is another local Ethernet-like network, the wider address of the next hop is still not enough to send a frame. G1 needs the local address of R1 on that link.

This brings us back to the machinery from Part 1. If the mapping is not already known, G1 asks the local network which MAC address belongs to the next-hop IP address. In IPv4 over Ethernet, this is ARP.

The forwarding process is therefore not one mysterious act called routing. It is a sequence of smaller questions:

Where is the packet ultimately going?
        ↓
Which route best matches that destination?
        ↓
What is the next hop on that route?
        ↓
Which local address belongs to that next hop?
        ↓
Send one local frame

Routing chooses the direction. ARP, when the current link is IPv4 over Ethernet, supplies the local delivery address needed to take the next step.

The same packet, a new local frame

The route across our empire might look like this:

Tower 0A in Valley 1
        ↓
Gateway G1
        ↓
Eastern relay R1
        ↓
Gateway G2
        ↓
Tower 0B in Valley 2

On the first hop, the local frame is addressed to G1. After G1 opens that frame and chooses the next route, it creates a new local frame addressed to R1. Relay R1 repeats the process and creates another frame addressed to G2. Inside Valley 2, G2 finally creates a frame addressed to tower 0B.

Figure 3. The blue end-to-end destination remains V2.12. Each gold local delivery ends at one next hop, after which a new local frame is created for the following link.

The IP destination does not become G1, then R1, then G2. It remains V2.12, because that is the end of the journey. What changes is the local delivery information wrapped around it.

This distinction gives us the mental model for the whole article:

The IP packet describes the end-to-end conversation. The local frame describes only the next delivery.

Every gateway repeats the same small procedure. It removes the local frame, reads the destination IP, consults its routing table, chooses a next hop, and creates whatever local frame the next link requires.

The routes also describe valleys rather than every individual tower. G1 does not need separate rules for V2.1, V2.2, V2.3 and every other address in Valley 2. One rule for V2.* is enough, because the next useful step is the same for all of them.

Our empire may eventually grow too large for routes to be written by hand. Routers can then exchange information and learn routes from one another, but that changes how the parchment is produced, not how a packet uses it. Forwarding still means reading the destination, selecting the best route and taking one local step.

We have now rebuilt the complete mechanism with towers. Local names move frames inside a valley. Wider addresses identify destinations across valleys. A boundary tells a tower whether the destination is local. A gateway accepts everything beyond that boundary. A routing table chooses the next hop, and each local network carries the packet one step farther.

Our little kingdom is no longer one network.

It is becoming a network of networks.

In other words, we have started building Layer 3.

Only now are we ready to leave the towers.

Back to the real world

Imagine one Linux machine connected to a home or office network:

IP address:      192.168.1.23/24
Interface:       eth0
Default gateway: 192.168.1.1
MAC address:     3c:52:82:xx:xx:xx

This is almost exactly what our tower knew. The machine has a local Ethernet identity, a wider IPv4 identity, a description of its local boundary, and the address of a gateway that can accept packets leaving that boundary.

Suppose an application wants to reach 192.168.1.50. Linux has a directly connected route for the local network, so the selected next hop is the destination itself. If the neighbor cache does not already contain the answer, ARP asks:

Who has 192.168.1.50?

Once the machine learns a mapping such as 192.168.1.50 → aa:bb:cc:dd:ee:ff, it can place the IP packet inside an Ethernet frame addressed directly to that MAC address.

Nothing particularly dramatic happened. We stayed inside one valley.

Leaving the valley

Now the same application wants to reach 8.8.8.8.

Linux exposes the relevant routes with ip route:

default via 192.168.1.1 dev eth0
192.168.1.0/24 dev eth0 scope link

At first this looks like the usual Linux habit of turning a simple idea into compressed hieroglyphics, but the table contains only two important rules. Destinations in 192.168.1.0/24 are directly reachable through eth0. Everything for which no more specific route exists goes through 192.168.1.1.

For 8.8.8.8, the local route does not match, so the default route selects 192.168.1.1 as the next hop. The packet remains addressed to 8.8.8.8, while the first Ethernet frame is addressed to the MAC address of 192.168.1.1.

If that MAC address is unknown, ARP asks:

Who has 192.168.1.1?

There is one technical correction worth making explicit here. It is common to explain the host as if it first performs a separate subnet test and then, only for a remote destination, remembers that routing tables exist. The operating system actually performs a route lookup. Configuring 192.168.1.23/24 on eth0 creates the directly connected route, while configuring the gateway creates the default route. Local and remote delivery are two outcomes of the same lookup.

Figure 4. The route lookup happens first. It selects either the destination itself or the gateway as the next hop, and only then does ARP resolve that selected next hop on Ethernet.

This ordering prevents two common confusions. ARP does not decide whether a destination is local, and it does not choose a route. It answers a local address-resolution question after routing has already chosen the next hop.

Where does the valley end?

Our tower used a rule such as V1.* to recognize its valley. IPv4 represents the same idea with a network prefix.

An IPv4 address contains 32 bits, which we usually write as four decimal octets. In 192.168.1.23/24, the /24 says that the first 24 bits belong to the network portion and the remaining 8 belong to the host portion.

For this address, the network is 192.168.1.0/24. Addresses such as 192.168.1.5, 192.168.1.50 and 192.168.1.200 belong to that network, while 192.168.2.5 does not.

The traditional subnet mask 255.255.255.0 expresses the same boundary. In binary it contains 24 ones followed by 8 zeroes. The ones mark the network portion, while the zeroes leave room for host addresses.

The boundary does not have to respect the dots in the decimal notation, because the dots exist for us and the bits exist for the machine. Consider 192.168.1.10/26. The first 24 bits and the first 2 bits of the final octet belong to the network, leaving 6 host bits.

Figure 5. Each final octet contains exactly eight bit positions. A /24 leaves positions 1 through 8 for hosts, while a /26 uses positions 1 and 2 for the network and leaves positions 3 through 8, exactly six bits, for hosts.

Those four /26 networks begin at:

192.168.1.0/26
192.168.1.64/26
192.168.1.128/26
192.168.1.192/26

This is why 192.168.1.10 and 192.168.1.50 belong to the first /26, while 192.168.1.70 belongs to the second. The decimal addresses look almost like neighbors, but the prefix has placed a real border between them.

The prefix is not an intermediate technique unrelated to our story. It is the exact mathematical form of the question our first tower had to answer:

Does the destination belong to my local network, or must the selected route lead through a gateway?

Who gave the host its map?

So far our computer somehow knew its IP address, its prefix, its default gateway and usually a DNS server. On a normal home or office network, nobody sits down every morning and manually configures every laptop, phone, printer and coffee machine that has somehow acquired WiFi.

Most of the time, DHCP does the paperwork.

A new host already has its local Ethernet identity, but it does not yet know its IPv4 address, its network boundary or its default gateway. It cannot send a normal IP request to a server whose address it does not know from an address it does not have, so it begins with a local broadcast:

I am here. Can somebody tell me who I am supposed to be?

The basic exchange is usually remembered as DORA: Discover, Offer, Request and Acknowledgement. The server may eventually give the client:

IP address:      192.168.1.23
Prefix:          /24
Default gateway: 192.168.1.1
DNS server:      192.168.1.1
Lease:           24 hours

DHCP did not route anything. It simply handed the host the small piece of the map it needs before routing can begin.

The lease means that the address is borrowed rather than carved into stone. Before the lease expires, the client normally tries to renew it, and the server may allow the same address to remain or assign another one.

The first DHCP discovery is a local broadcast, and routers normally do not forward local broadcasts. When the DHCP server lives in another network, a DHCP relay listens locally and carries the request toward the remote server. This lets one server configure many networks without joining their broadcast domains together.

If DHCP is absent, nothing fundamental breaks. We can configure the IP address, prefix, gateway and DNS server manually, provided that the values are correct and the chosen address does not conflict with another host.

Small networks often place DHCP, routing, NAT, firewalling, DNS forwarding and WiFi inside the same plastic box near the wall. Calling that entire creature a router is convenient but slightly misleading. Router is one role performed by the device, just as a castle can be a home, a fortress, a court, a prison and a very expensive heating problem at the same time.

Following one real packet

Host A lives at 192.168.1.23/24, and somewhere beyond several networks Host Z lives at 10.20.0.42/16. Host A’s route lookup selects 192.168.1.1 as the gateway, and ARP gives it the gateway interface’s MAC address.

The word interface matters here. A router connects at least two networks, so it does not have one universal IP address or one universal MAC address. Each Ethernet interface belongs to a particular local network and has its own MAC address, while it will usually have an IP address in that network as well.

Figure 6. A router’s interfaces belong to different local networks. The incoming frame ends at Interface A, while a new local frame leaves through Interface B with that interface’s identity.

Hosts on 192.168.1.0/24 use 192.168.1.1 as their gateway because that address belongs to the router interface standing inside their own network. The interface on the other side belongs to another network and has different local neighbors.

Host A creates the first frame:

Ethernet destination: MAC of 192.168.1.1
IPv4 destination:     10.20.0.42

The router removes the Ethernet frame, reads the IPv4 destination, performs its own route lookup and chooses the next hop. If the outgoing link is also Ethernet, the router resolves that next hop’s MAC address and creates a completely new frame. The next router repeats the same process.

Eventually a router finds that 10.20.0.0/16 is directly connected. On the final Ethernet network, it uses ARP to learn Host Z’s MAC address and creates the last local frame.

The IPv4 source and destination normally remain the same in this simple routed topology. NAT can deliberately rewrite addresses, but NAT is another official with another stamp and is not required to explain routing.

The packet is not completely untouched, however. Every router decreases the IPv4 Time to Live, usually called TTL, by at least one. Because TTL is part of the IPv4 header, the router also updates the IPv4 header checksum.

Figure 7. The IPv4 source and destination stay end to end, each local frame is rebuilt for one link, and TTL decreases as routers forward the packet.

TTL exists because routing information can be wrong. If two routers keep sending a packet back and forth, the packet must not wander between them until the end of history. When TTL reaches zero, the router discards it and normally sends an ICMP Time Exceeded message back toward the source.

Ethernet is not mandatory on every link. Another link technology may use different framing and may have no MAC addresses or ARP at all. What survives is the larger mechanism: the router removes the old link-layer envelope, keeps the IP destination, chooses the next route and creates whatever local delivery the outgoing link requires.

Layer 3 does not bypass Layer 2. It uses one local link, then another one, then another one. A global network is made from local networks that repeatedly agree to pass the packet on.

Routers do not know the whole journey

A router does not normally calculate the complete road from Host A to Host Z. It needs to answer only one question:

Where should I send this packet next?

This is why routing tables usually describe networks rather than every individual host. One rule can say that the whole 10.20.0.0/16 network is reachable through Router B.

Sometimes several routes match the same destination:

10.0.0.0/8       via Router A
10.20.0.0/16     via Router B
10.20.5.0/24     via Router C
default          via Router D

The address 10.20.5.17 matches every line, including the default, but the /24 route wins because it has the longest matching prefix. A longer prefix describes a smaller and more specific group of addresses. The default route has a prefix length of zero, so it is the vaguest possible instruction: if nobody knows better, send the packet this way.

For a small network, somebody can configure these routes by hand. In a much larger system, routers can exchange routes through protocols such as OSPF, IS-IS and BGP. Those protocols deserve their own story, but forwarding remains the same after the table has been built: read the destination IP, choose the most specific matching route, select the next hop, and perform one local delivery.

What actually happens when we type ping

When we run:

ping 8.8.8.8

the operating system performs the route lookup, selects the next hop, resolves the necessary local address if the outgoing link requires it, and sends an IPv4 packet containing an ICMP Echo Request.

If the remote host chooses to answer, it sends an ICMP Echo Reply back toward us. A successful reply means that the forward route worked, the return route worked, the necessary local deliveries succeeded, and the policies along the way allowed ICMP.

A failed ping therefore does not prove that the destination is dead. It proves only that the complete conversation did not return, which is a much less satisfying answer and a much more honest one.

The mental model to keep

We began with a local network that knew only local tower names. The second valley made those names ambiguous, so we added a wider address that named both the valley and the tower. Once a destination could be remote, we needed a boundary that distinguished our local world from everything beyond it. A remote packet needed a gateway, and the gateway needed a routing table that selected the next useful step.

Every new idea appeared because the growing system made the previous model insufficient.

The result can be carried in one sentence:

Routing keeps an end-to-end IP destination inside the packet while rebuilding the local delivery around it for every link.

A MAC address answers who should receive this frame here. An IP address answers where the packet is ultimately going. A prefix describes which addresses belong to a network. A route chooses the next hop, and ARP maps that next-hop IP address to a local Ethernet address when the current link needs one.

None of this replaced the network from Part 1. We built above it and then reused it at every hop.

Put those answers together and our valleys become a network of networks, which is rather inconveniently the same direction in which our rulers were taking their empire anyway.

One machine is still a black box

There is one character in this story that we have treated with suspicious casualness: the computer itself.

Throughout both parts I have written that the computer checks a route, stores a neighbor, owns a MAC address and sends a frame, but what inside the machine actually does those things? Where does a network interface begin and the operating system end? What changes when one computer has several interfaces, and what happens when some of those interfaces are not physical at all?

Linux can create virtual interfaces, connect them with virtual Ethernet pairs, place them in bridges, and split the network into namespaces that behave like separate hosts. One physical machine can contain several isolated valleys and the roads between them.

Which means that, instead of invading another kingdom for the next experiment, we can build almost the entire empire inside one Linux machine.

That is where we go next.

Series Hub:

Recently I started working on a networking series for software developers who enter the electrical-grid world and suddenly discover that the grid speaks in packets, interfaces, switches, routes, protocols and a suspicious number of acronyms.

Before going anywhere near IEC 61850, utility WANs or substation networks, I wanted to understand the ordinary networking machinery itself. Not only what a switch or ARP does, but why these things had to appear in the first place.

I could start with the OSI model. I have seen that diagram enough times in my life. The problem is that a finished diagram is very good at telling us where something belongs and surprisingly bad at showing why it exists.

So I had a more ridiculous idea.

Imagine that I am somewhere in medieval Europe. No copper cable, no fiber, no radio and definitely no Ethernet. I have stone towers on hills, and the only useful property I can rely on is visibility. If one tower can see another, I can make light visible or hide it.

Could I start from that and slowly rebuild a network?

This is not a historical reconstruction. I am not claiming that medieval Europe was one good standards committee away from IEEE 802.3. The towers are only a constraint. I want to begin with the smallest possible communication system, push it until it becomes uncomfortable, fix the new problem, and then compare the thing we invented with the machinery inside a real network.

That approach is much closer to how I like to understand systems. The finished technology often hides the reasons for its own shape. Broken and incomplete versions expose them.

In this post we will discuss the local part of the network. We will begin with physical signalling between two towers, then add more participants and see why frames, MAC addresses, switching, flooding, broadcast and ARP appear. Near the end we will come back to Linux and look at the same machinery on a real machine.

Subscribe now

Two towers and one bit

I want to begin before packets, addresses and protocols exist at all.

Two towers stand on neighboring hills. Tower A can see Tower B directly.

Diagram 1. Our entire network consists of two towers with direct visibility.

Tower A wants to send one piece of information to Tower B. The bridge is closed, for example.

But even that sentence is already too advanced for our network. We do not yet have a way to move a single bit between the towers.

The simplest thing I can invent is a lantern with a shutter. During a fixed interval the light is either visible or hidden.

light visible = 1
light hidden  = 0

So a guard can produce something like:

1 0 1 1 0 0 1 0

And the other tower can observe it.

This looks almost stupidly primitive, but it already contains the first real networking problem. An abstract 1 inside one system has to become something physical, travel through a medium, and be recovered by the other system.

In our case the medium is light through air. In Ethernet it may be an electrical or optical signal. Wireless uses radio. Real physical layers do not literally push abstract bits through a cable one after another. They deal with symbols, encoding, timing, synchronization, noise, attenuation, clock recovery and many other details that become a rabbit hole very quickly.

Even the lantern version immediately asks awkward questions. How long does one interval last? How does Tower B know where the sequence begins? What happens in fog? If the light stays visible for three intervals, how does the receiver know that it saw 111 rather than one long 1?

I am going to stop before our medieval guards have to invent signal processing.

The useful part for this article is that we have reconstructed the job of Layer 1: make information physical enough that it can cross one link and be recovered at the other side.

That works beautifully while our entire civilization consists of two towers.

Unfortunately, I want a network.

Adding the rest of the valley

So I add more towers.

Some can see each other directly. Some need a relay point between them. For the moment, imagine one local communication region where a relay tower has direct light paths toward several participants.

Diagram 2. Several towers now share one local communication system through a relay point.

The light can move bits across each individual path, but the moment several towers share the system, a stream like this becomes annoyingly ambiguous:

10110010...

Who is it for? Where does one message end and the next one begin? If the relay receives it, which path should it use? And if I later want to carry different kinds of data, how does the receiver know what is inside?

This is where the simple Layer 1 picture starts to break. The physical signal still matters, but I need structure above it.

Give the towers local addresses

I will start with the most obvious missing piece and give every tower a local identifier.

Tower A = 0A
Tower B = 0B
Tower C = 0C
Tower D = 0D

There is nothing magical about these values. My tiny kingdom does not need a 48-bit address space yet.

Now instead of sending naked bits, Tower A can send a structure:

TO:   0C
FROM: 0A
TYPE: MESSAGE
DATA: THE BRIDGE IS CLOSED

The relay no longer has to guess what the bits mean. There is a destination, a source, a description of what the payload contains, and the payload itself.

That structure is already suspiciously close to an Ethernet frame.

Our tower frame Ethernet equivalent What it is doing TO Destination MAC address Identifies the local recipient of the frame FROM Source MAC address Identifies the local sender of the frame TYPE EtherType Tells the receiver what kind of payload is carried DATA Payload Carries the higher-level data

Table 1. The fields I needed for the tower network have direct relatives in an Ethernet frame.

A real Ethernet frame has more machinery around this, and the signal on the wire has additional synchronization and encoding details. Our invented text header is obviously not its real wire format. Those details solve different problems, so I will leave them outside this model for now.

The important thing is that 0A, 0B and the rest are local link-layer identities. They play the role of MAC addresses. They do not tell us how to reach a tower somewhere on the other side of Europe. They let this local network move a frame toward the right participant.

This is roughly where Layer 2 enters the picture. We have stopped dealing only with signals and started dealing with frames and local delivery.

The relay starts remembering

The first version of my relay can be extremely lazy. Every time it receives a transmission, it can repeat the signal toward every other light path. At the frame level, the result is that every frame becomes visible everywhere.

This is roughly the world of an old Ethernet hub. A real hub operates at Layer 1 and repeats signals rather than parsing Ethernet frames, which is one place where the tower analogy should not be pushed too literally.

But the relay is sitting in a very useful position. When a frame arrives from the western path and says:

FROM: 0A

it has learned something for free: whatever 0A is, the western path is how I just heard from it.

So the relay can remember that fact.

After some traffic its memory may look like this:

0A → western path
0B → northern path
0C → eastern path
0D → southern path

Now a frame addressed to 0C does not have to be repeated everywhere. The relay can send it only toward the eastern path.

This is the part where our relay has quietly become an Ethernet switch.

A switch builds a forwarding table by looking at the source MAC address of frames arriving on its ports. The source address is evidence. If a frame from 0C arrived through port 3, then 0C is currently reachable through port 3. Real switches also age these entries because networks change, cables move and machines disappear.

The destination MAC is used for the next decision. If the switch already knows where that destination lives, it forwards the frame through the corresponding port.

The destination that has never spoken

This was the first place where I caught myself giving the relay knowledge it had never earned.

Suppose its table contains only:

0A → western path
0B → northern path

Then a frame arrives from the west:

FROM: 0A
TO:   0C

The relay can refresh 0A → western path, because it has just observed that fact. But the destination field tells it absolutely nothing about where 0C is located. TO: 0C only describes what the sender wants.

So the relay has one honest option: try all the other paths.

It retransmits the frame everywhere except back through the path it arrived on. Every participant may see it, but only 0C should accept a unicast frame addressed to 0C.

Ethernet calls this unknown unicast flooding.

If Tower C later sends anything:

FROM: 0C
TO:   0A

then the relay finally gets real evidence. The frame came from the east, so it can learn:

0C → eastern path

Future frames for 0C can go directly there.

There is a subtle point here that I want to keep separate from ARP. Unknown unicast flooding means the sender already put a destination MAC address into the Ethernet frame, but the switch does not know which port leads to that MAC. ARP solves a different problem: the sending host may not know the destination MAC address at all.

In a normal fresh IPv4 conversation, ARP often causes the destination host to reply before the first IP unicast frame is sent. That reply also gives the switch a chance to learn the destination MAC. So you may not actually observe unknown unicast flooding every time you contact a new local IP address.

The behavior still matters. Imagine that Tower A already knows 0C, perhaps from an existing neighbour-cache entry, while the switch’s own forwarding entry for 0C has expired. Then A can send a perfectly valid unicast frame to 0C, and the switch may still have to flood it.

Diagram 3. The relay tower has become a Layer 2 switch by learning where local addresses are reachable.

The switch now has a useful memory, but Tower A can still be stuck for a completely different reason.

I know the IP address. I still cannot send the frame

Suppose an application on Tower A wants to send data to:

192.168.1.20

Let us say that this address belongs to Tower C.

For the moment I will also grant Tower A one piece of knowledge: it knows that 192.168.1.20 is on the same local network. How the host makes that decision belongs to IP addressing, subnet masks and routing tables, which is exactly the problem I want to pick up in the next part.

Tower A can create an IP packet for 192.168.1.20, but our local medium still does not forward an ordinary Ethernet frame by looking at that IP address. The Ethernet frame needs a destination MAC address.

And Tower A does not know it.

This is different from the problem we just had inside the switch.

Missing knowledge Who is missing it? Mechanism MAC → switch port The switch Source-MAC learning, with unknown unicast flooding when the destination is unknown IPv4 address → MAC The sending host ARP

I like this distinction because both failures can sound like “I do not know where C is,” while they live in different machines and are solved by different mechanisms.

ARP, the Address Resolution Protocol, exists to solve the second one for IPv4.

A local shout

There is a bootstrapping problem, though. If Tower A does not know Tower C’s MAC address, it cannot send Tower C a normal unicast Ethernet frame asking for it.

So I need one more local address in the tower world:

EVERYONE

A frame sent to EVERYONE is copied toward every participant in this local communication region.

Ethernet already has exactly such a destination:

ff:ff:ff:ff:ff:ff

That is the Ethernet broadcast MAC address.

Tower A can now send something equivalent to:

Who has 192.168.1.20?
Tell 192.168.1.10.

The Ethernet destination is the broadcast address, so the switch floods the frame through the local broadcast domain. The switch does not need to understand the IP address in the question. In the ordinary forwarding model it is still doing a Layer 2 job: it sees a broadcast destination MAC and forwards accordingly.

Inside that Ethernet frame is an ARP message. ARP is not an IP packet, and the switch does not use ARP to decide where ordinary unicast Ethernet frames should go. ARP is a separate protocol carried directly inside Ethernet, identified by EtherType 0x0806.

Every IPv4 host in the local broadcast domain may receive the request. Most of them inspect it, notice that 192.168.1.20 is not theirs, and move on with their lives.

Tower C recognizes its own IP address and replies with its MAC address:

192.168.1.20 is at 02:00:00:00:00:0c

The reply can be sent as a unicast Ethernet frame back to Tower A because the ARP request already contained A’s addresses.

That reply does two useful things in two different places. Tower A can store:

192.168.1.20 → 02:00:00:00:00:0c

in its neighbour cache, while the switch sees a frame arriving from Tower C and can learn 02:00:00:00:00:0c → eastern port from the source MAC address.

Nobody had to teach the switch about 192.168.1.20. It still does not need that information for normal Layer 2 forwarding.

Diagram 4. Broadcast lets ARP discover the local MAC address associated with an IPv4 address.

This is the part of the experiment I find especially satisfying because we now have two small pieces of memory that look similar on paper but belong to completely different parts of the system.

Two different maps of the same local network

Tower A may remember:

192.168.1.20 → 02:00:00:00:00:0c

That is host knowledge. On Linux it lives in the neighbour table. For IPv4, ARP is the mechanism that normally creates and refreshes this mapping.

The switch may remember:

02:00:00:00:00:0c → eastern port

That is switch knowledge. It was learned from the source MAC address of a frame that arrived through the eastern port.

The two tables answer different questions:

Machine What it knows Why it needs it Host IPv4 address → MAC address To build an Ethernet frame for a local IPv4 destination Switch MAC address → port To decide where to forward that Ethernet frame

This also makes the MAC and IP distinction less mystical.

A MAC address is useful for delivery on the current Layer 2 network. An IP address is the logical address of the packet’s destination in the larger Layer 3 system.

When A and C are sitting in the same valley, both addresses appear to identify the same machine, so having two of them can feel redundant. The redundancy is mostly an illusion created by the small example. Once the packet has to cross a router, the reason for both becomes much clearer.

For a local destination, A may build something conceptually like this:

Ethernet:
  dst MAC = 02:00:00:00:00:0c
  src MAC = 02:00:00:00:00:0a
  type    = IPv4

IPv4 packet inside:
  dst IP  = 192.168.1.20
  src IP  = 192.168.1.10

The switch cares about the Ethernet destination MAC when forwarding the frame. Tower C eventually receives the frame, sees that the payload is IPv4, and passes the enclosed IP packet upward.

The complete first exchange is therefore not one magic lookup. It is several small mechanisms cooperating:

Application wants to send to 192.168.1.20
            ↓
Host decides 192.168.1.20 is local
            ↓
Neighbour cache has no MAC for 192.168.1.20
            ↓
Host sends an ARP request in an Ethernet broadcast frame
            ↓
Switch learns A's source MAC and floods the broadcast
            ↓
C recognizes its IP address and sends an ARP reply
            ↓
Switch learns C's source MAC from the reply
            ↓
A stores 192.168.1.20 → C's MAC in its neighbour cache
            ↓
A puts the IP packet into a unicast Ethernet frame for C's MAC
            ↓
Switch uses C's MAC to select the correct port
            ↓
C receives the frame and extracts the IP packet

This is a simplified path, but it is finally a simplification I can reason with. Each table has an owner. Each address has a purpose. The switch never needs to resolve 192.168.1.20, and ARP never tells the switch which destination port to use.

Back on a real Linux machine

After spending enough time with imaginary towers, I wanted to see whether the same structure is visible without the analogy.

Linux exposes the host-side neighbour table with:

ip neigh

For an IPv4 neighbour, an entry can look like this:

192.168.1.20 dev eth0 lladdr 02:00:00:00:00:0c REACHABLE

There it is: the same IPv4 address → MAC address mapping that Tower A had to learn. The Linux neighbour table is more general than ARP and also contains IPv6 neighbour information, but for IPv4 on Ethernet, ARP is the mechanism we care about here.

The exchange itself is even nicer because we can watch it happen:

sudo tcpdump -i eth0 -n -e arp

If eth0 is not your actual interface name, which is very likely on a modern Linux machine because apparently eth0 was too easy, replace it with the real interface.

With no cached neighbour entry, contacting 192.168.1.20 should first produce an ARP request sent to the Ethernet broadcast address. Then the owner of that IP replies with its MAC address. After the mapping exists, the host can send normal unicast Ethernet frames.

If the neighbour entry already exists, the absence of ARP traffic is not evidence that the mechanism disappeared. Linux is simply using what it already knows.

🔎 Try it yourself: run ip neigh, start sudo tcpdump -i <interface> -n -e arp, and then ping another machine on the same local network. If a cached entry prevents the ARP exchange from appearing, remove only that entry with sudo ip neigh del <IP> dev <interface> and repeat the experiment. Use a machine and network you control.

I will stop the Linux part here. Later in Act I I want to open one host properly and look at interfaces, bridges, network namespaces, virtual Ethernet devices and the rest of the machinery Linux piles underneath an innocent socket. Here I only wanted to verify that our medieval invention leaves fingerprints on a real machine.

Then I add another valley

So far everybody belongs to one local communication region. Broadcast can reach the whole group, and one Layer 2 system is enough.

Then I put another valley behind the mountains.

It has its own towers, its own local relay, its own broadcasts and its own local MAC addresses. Tower A cannot simply shout an ARP request across the mountains and expect all of Europe to participate.

Still, Tower A needs to send a message there.

Diagram 5. The local-network model stops being sufficient when the destination belongs to another network.

This is where my local-network model finally runs out of road.

I could try to connect both valleys into one gigantic Layer 2 domain and let broadcasts spread farther and farther. Then I could keep adding valleys until one ARP request lights half the continent. Apart from being a rather beautiful medieval spectacle, it would also turn every local broadcast into everybody else’s problem.

The cleaner solution is to let each valley remain a local network and introduce a machine that knows how to move packets between networks.

That machine is a router.

And here the two-address idea starts paying rent.

If Tower A wants to send an IP packet to a host in the remote valley, the packet can keep the remote host as its IP destination. But the Ethernet frame used in A’s local valley is not addressed to that distant machine. It is addressed to the local router, the next machine that can move the packet closer to where it belongs.

Conceptually:

IP destination:       remote host in another valley
Ethernet destination: local router

The router receives the local frame, removes that Layer 2 envelope, examines the IP packet, chooses the next route, and places the packet into a new Layer 2 frame appropriate for the next link.

This is the point where the medieval analogy also needs a new piece. A relay inside one valley was enough while everybody shared one local addressing and broadcast system. Communication between valleys requires a gateway that understands something larger than local tower addresses.

That larger system is what I want to rebuild in Part 2: IP addresses, subnet masks, the decision that a destination is local or remote, default gateways, routing tables and routers.

The towers got us surprisingly far, but the mountains finally forced us into Layer 3.

Which we cover in the next part!

Stay tuned!

Series Hub

References and further reading

• RFC 826: An Ethernet Address Resolution Protocol

• IEEE 802.3 Ethernet Working Group

• Linux ip-neighbour manual

• Linux kernel TUN/TAP documentation, useful later when we rebuild networking machinery inside one Linux host

• A Developer’s Map of the European Power Grid, the wider series context for software developers entering the energy domain

I have worked with networked software for years, but for most of that time the network was something I used rather than something I really understood. I could open a socket, call an API, connect to an IP address, and continue with the part of the system I was responsible for.

When I started working closer to the power grid, this became uncomfortable. Network details were no longer hidden somewhere below the application. Ethernet frames, multicast, VLANs, gateways, industrial protocols, and utility networks appeared directly in requirements and discussions.

The difficult part was not learning what each term meant. I could read a definition of ARP or a description of a routing table. The problem was seeing how all these mechanisms formed one system, and then seeing how that system was used by substations, protection devices, control centres, and remote engineering tools.

This series is my attempt to build that connected picture. I want to begin with ordinary network machinery, reconstruct enough of it in Linux to make it tangible, and then follow it into the power grid.

In this post I will explain the plan and keep links to every part as it is published. The plan may change while I work through it, and this page will change with it.

Subscribe now

Why networking in the power grid needs its own map

While writing A Developer’s Map of the European Power Grid, I kept running into communication at every level of the system. The last article, Where Software Lives in the Power Grid, made the gap even more obvious. A relay, an HMI, a gateway, and a control-centre service are all pieces of software, but the networks around them are shaped by very different physical jobs.

The vocabulary is dense, although the vocabulary itself is not the hardest part. The real difficulty is connecting ordinary mechanisms such as switching, routing, multicast, and redundancy to the events they carry. A frame in a substation might represent a protection decision. A remote connection might give an engineer access to equipment that controls real power.

When the networking foundation is fuzzy, grid protocols can look like a collection of arbitrary rules. When the grid context is missing, a generic networking explanation can feel detached from the work. I want this series to bring those two views together for software developers who are entering the energy domain.

How I want to approach the subject

I learn technical systems best when I can take a small version apart and rebuild it. That is the approach I want to use here.

Instead of beginning with a chapter called “ARP,” I would rather begin with two machines on one network and ask how one finds the other. Instead of introducing routing as a table of definitions, I want to place the destination somewhere else and watch the sending host decide that the packet has to leave through a gateway. The mechanism arrives because the problem makes it necessary.

Linux is a useful laboratory for this. Network namespaces can behave like separate hosts, veth pairs can connect them, a bridge can act as a switch, and another namespace can forward packets as a router. These experiments are small enough to run on one computer, but they expose the same decisions that appear in larger networks.

❓The question I want to keep asking is simple: what problem made this mechanism necessary?

The examples will be simplified on purpose. They are models for understanding packet paths and engineering choices, not reference designs for operational substations or utility networks.

The three acts

I currently see the series in three acts. Each one widens the system under investigation: first one host and its nearby network, then a substation, and finally the connections between substations, control centres, and remote users.

Diagram 1. Each act reuses the mechanisms established in the previous one and applies them to a wider part of the grid.

The division is useful for planning, although I do not consider it fixed. If an experiment shows that an article belongs somewhere else or deserves its own part, I will change the map.

Act I: Networking machinery for software engineers

Act I is where I fill the gaps in my own networking foundation. We start with the smallest useful network: two machines, a physical link, and a switch. From there we can see how Ethernet frames move, how hosts learn MAC addresses, and why ARP exists.

The first part begins at Layer 1, or L1. Copper and fibre carry electrical or optical signals, while Ethernet gives those signals a frame structure and an addressing model at Layer 2. I do not plan to turn this into an electronics course, but the physical layer matters because every packet eventually depends on a real medium, a negotiated link, and hardware that can fail.

Next, we break the local-network assumption by placing the destination somewhere else. This introduces gateways and routing for a concrete reason. Only after those ideas are visible from the outside do we open a Linux host and find the interfaces, neighbour table, routes, bridges, namespaces, and virtual links that implement them.

Linux comes last for a reason. Once the external network is clear, a single computer can reproduce it. Namespaces become hosts, veth pairs become links, a bridge becomes a switch, and a forwarding namespace becomes a router. That gives us a laboratory for the rest of the series.

Part 1: How Local Networks Work

Layer 1, Ethernet, MAC addresses, switches, and ARP

We will connect two hosts on one local network and follow a message from the physical link upward. The main question is deceptively simple: what has to happen before one machine can send an Ethernet frame to another?

Part 2: How Packets Travel Between Networks

IP addresses, subnet masks, gateways, routing tables, and routers

The destination now moves outside the local network. We will follow the packet through the host’s routing decision, the default gateway, and the router that connects the two sides.

Planned.

Part 3: How Linux Builds a Network

Interfaces, neighbour tables, routes, namespaces, veth pairs, bridges, forwarding, and TUN/TAP

We will rebuild the first two parts inside one Linux machine and inspect the objects that the kernel uses to represent the network.

Planned.

Act II: Networking inside the power grid

Once the ordinary machinery is clear, I want to bring it into a substation. This is where familiar networking ideas begin carrying unfamiliar consequences. Ethernet and multicast are still Ethernet and multicast, but now the traffic can describe measurements, status changes, or protection events.

A small substation model should make this easier to reason about. We can place an IED, an HMI, a gateway, and a station bus on one diagram, then ask why the network is segmented, which traffic is multicast, where time matters, and what happens when a link or switch fails.

Articles I currently plan

Building a Tiny Substation Network

A compact network with representative devices, traffic paths, VLAN boundaries, and the practical reasons behind them.

Why IEC 61850 Is Not One Protocol

A map of the different jobs hidden behind one standard, including configuration, client-server communication, fast events, and sampled measurements.

Rebuilding GOOSE From First Principles

An attempt to understand GOOSE by starting with the protection problem, then examining multicast delivery, repeated messages, timing, and failure behaviour.

Act II is planned. I will add links here as the articles are released.

Act III: Connecting the grid

A substation is only one island. The next step is to follow communication beyond it: toward a control centre, another site, or an engineer connecting from somewhere outside the operational network.

This is where routing, firewalls, wide-area links, tunnels, and trust boundaries meet. A diagram that says “remote access” can hide a surprising amount of machinery, so I want to open that path and inspect each boundary.

Articles I currently plan

From a Substation to a Control Centre

We will follow telemetry and commands across a simplified utility topology and identify the role of each network segment.

How Telecontrol Crosses Networks

This part will connect application protocols to TCP/IP, routing, firewalls, and operational constraints without treating the network as an invisible pipe.

Building a Secure Tunnel Into a Substation

A small WireGuard experiment will show what a tunnel actually creates, which routes change, and where encrypted traffic enters and leaves the system.

Act III is planned. I will add links here as the articles are released.

What this series should make possible

My target is fairly modest: I want the network below an application to become something I can inspect and reason about.

When I see an unfamiliar architecture, I want to identify the local-network boundaries, understand why a packet follows a particular gateway, recognise where multicast is being used, and know which Linux tools can confirm my assumptions. In a grid diagram, I also want to connect those mechanisms to the physical role of a relay, gateway, substation, or control centre.

That level of understanding will not make me a protection engineer or a utility network architect. It should, however, make the conversations with those specialists much more useful and make network-related failures less mysterious.

Follow the series

I will use this page as the index for the whole project. As each article is published, I will replace its status with a direct link and update the plan if the investigation takes a different direction.

I am learning this subject in public. If you work with industrial networks, IEC 61850, utility communications, or Linux networking and notice that I have missed something, I would be glad to hear from you.

Previous part:

“The metasphere remains. But it is more wilderness than ever now. Black forests of unknown time and space. Sounds in the night. Lions. And tigers. And bears.” ― Dan Simmons, Hyperion

Where does software live in the grid?

It lives between, within and above.

For most of this series, I was looking at the physical world. Generators produce electricity, transformers change voltage, transmission lines connect regions, and substations switch and protect all of it. These things have weight, occupy space and can be found on a map.

But during my research I slowly realised that there is another world hiding inside this one.

Software lives inside protection relays and control-room servers. It lives between devices, companies and countries. It stretches above cables and transformers as measurements, alarms, schedules, forecasts, bids and models. Almost every hollow space that is not fulfilled by the physical world hides another piece of software.

And it does not only observe the industry. It moves it. Software opens breakers, reconstructs the state of the grid, predicts tomorrow’s demand, accepts bids, calculates prices, activates reserves and later calculates who owes money to whom because reality did not follow the plan.

For me, it looks like a huge augmented reality built on top of the grid. At some point it becomes difficult to decide which reality is primary. Electricity still follows physics, but more and more of what happens around it is decided, coordinated and recorded in software.

It felt like discovering another continent.

I do not want to pretend that I can produce its Google Map. What I can draw is closer to the maps of Herodotus, collected from documents, conversations, tools I met at work and stories from people who travelled further than I did. It will contain blank places and probably a few sea monsters, but it should still be enough to understand where we are.

And AI, for better or worse, has been my Virgil during this journey.

In this post we will discuss three large software worlds: the software that operates the physical grid, the software that operates the electricity market, and the software that connects financial plans with physical reality.

Figure 1. Three software worlds of the grid. Their borders are not clean, but their main purposes are different.

Subscribe now

Software that operates the physical grid

This is the software closest to copper, steel and electromagnetic fields.

A breaker does not know that it is called Q0. A current transformer does not produce a neat object with a timestamp and a quality flag. The physical world gives us voltage, current, heat, motion and consequences. Software turns them into information that people and other systems can use.

The first computers we meet are Intelligent Electronic Devices, usually shortened to IEDs. This family includes protection relays, bay controllers, merging units and meters.

A protection relay watches electrical quantities and reacts when they indicate a fault. It may detect overcurrent, an abnormal frequency or a fault inside a transformer and send a trip command before a remote operator could even understand what happened.

A bay controller supervises breakers, disconnectors and earthing switches around one part of a substation. A merging unit digitises current and voltage measurements. A meter measures energy or power for operational and commercial use.

Above these devices sits the substation automation system. Its visible part is often a local HMI with a single-line diagram, measurements, alarms and controls. Behind it are gateways, Remote Terminal Units (RTU), event recorders, engineering tools and configuration files. The substation is already a distributed software system, only with unusually expensive outputs.

Communication inside a modern substation is often organised around IEC 61850.

At first, it is tempting to think of IEC 61850 as just another network protocol, similar to HTTP or Modbus. But it is much broader than that. It is a common language for describing the devices, measurements, commands and protection functions inside a power-system automation system.

For example, different vendors may build protection relays in very different ways, but IEC 61850 gives them a shared model for concepts such as circuit breakers, voltage measurements, alarms and trip signals. This makes it easier for devices and engineering tools from different manufacturers to understand each other.

IEC 61850 also defines several ways in which these devices communicate. MMS is commonly used when one system needs to read data from another device, receive reports or send a control command. GOOSE is used for very fast messages between devices, for example when one protection relay must immediately tell another device to trip a breaker. Sampled Values are used to send digitised current and voltage measurements over the station network.

The standard also includes SCL, or Substation Configuration Language. SCL files describe which devices exist in the substation, what data they provide, how they are connected and how they communicate. In other words, IEC 61850 defines not only how messages travel, but also what those messages mean and how the whole system is engineered.

Communication between a substation and a remote control centre often uses another standard called IEC 60870-5-104, usually shortened to IEC 104.

Its role is easier to imagine. A gateway or Remote Terminal Unit inside the substation gathers selected information from local devices and sends it to the control centre. This can include voltage and power measurements, breaker positions, alarms and events. The control centre can then send commands back, for example to open or close a breaker.

A simple mental model is this: IEC 61850 is often used to organise communication inside the substation, while IEC 104 is often used to connect the substation with the control centre.

There are many other protocols, including DNP3, Modbus and OPC UA. There are also decades of legacy equipment and vendor-specific behaviour, because standards can reduce chaos but apparently cannot eliminate human creativity.

At the control centre, SCADA collects telemetry, displays the current state, processes alarms and lets authorised operators issue commands. It is the operational window into the grid.

For transmission operators, SCADA is often surrounded by an Energy Management System, or EMS. The EMS runs applications such as state estimation, power-flow calculation and contingency analysis. State estimation combines measurements with a network model and calculates the most plausible state of the system. Contingency analysis asks what would happen if a line, transformer or generator disappeared.

Distribution operators use similar families of software, often called DMS or ADMS, to supervise distribution networks, manage outages, estimate states and increasingly coordinate distributed generation, batteries and flexible loads.

Around all of this sit historians, alarm systems, disturbance recorders, outage tools, switching-management systems and dispatcher training simulators. Each solves a different problem, but together they perform one purpose: they let humans observe, understand and control a physical system that is far too large and fast to operate directly.

Figure 2. A simplified path from physical equipment to the operator. Measurements travel to the right, while decisions and commands travel back.

Software that operates the electricity market

The market lives in a different world.

It does not normally open breakers or read raw current samples. It works with products, bids, schedules, contracts, positions and money. Still, it cannot completely escape physics, because the product it trades must be generated, transported and consumed at a specific time.

The process usually starts with forecasting and analytics. Generators forecast their available production. Suppliers and balance responsible parties forecast consumption. Traders forecast prices, weather and the behaviour of everybody else who is also forecasting prices, weather and the behaviour of traders. There is a pleasing circularity to the whole exercise.

These forecasts enter trading and bidding systems. Market participants prepare orders for forward, day-ahead, intraday and balancing markets. The software checks limits, builds bidding strategies, submits orders and records the results.

At the exchange side, market systems collect bids and offers. In the European day-ahead and intraday markets, market coupling allocates cross-zonal capacity together with energy orders. The Capacity Allocation and Congestion Management rules define the framework for this process. The algorithm does not simply find a buyer for every seller. It tries to maximise economic surplus while respecting the capacity available between bidding zones.

Once the market clears, the result becomes a schedule. Generators know how much they should produce, suppliers know what they bought, and balance responsible parties have a position that they must later defend against reality.

Large participants commonly use Energy Trading and Risk Management systems, or ETRM. These systems keep track of trades, contracts, portfolios, exposure, nominations, collateral and settlement. They are the commercial memory of the company.

After delivery, metering and settlement systems compare contracted quantities, schedules and measured energy. They calculate invoices, fees and later the cost of imbalances. The market therefore does not end when the price is published. It ends much later, when the final numbers agree enough that money can move.

Figure 3. The commercial loop. A forecast becomes a bid, the bid becomes a schedule, and physical delivery eventually returns as money and new data.

Software that connects both worlds

The third area is where the article becomes especially interesting for me.

A market participant can create a good schedule. A control centre can observe the real grid. Neither is enough alone. Somebody must continuously compare the planned world with the physical one and decide what to do when they differ.

This is one of the main software responsibilities of a Transmission System Operator.

A TSO receives schedules, forecasts and availability information from market participants. At the same time, SCADA and EMS show what the system is actually doing. The TSO compares the two realities and checks whether the grid can safely carry the planned flows.

This area contains load and generation forecasting, scheduling systems, network models, capacity-calculation tools, congestion-management tools and operational-security analysis. European TSOs also exchange their individual grid models and combine them into a Common Grid Model through shared infrastructure such as the Operational Planning Data Environment. It is software used not merely by one company, but to construct a common view of an interconnected continent.

When expected flows violate limits, TSOs may use remedial actions such as changing generation, modifying topology, redispatching power or countertrading. When production and consumption diverge in real time, balancing software procures and activates reserves.

Some control happens almost automatically. Frequency Containment Reserves respond quickly to frequency deviations. Automatic and manual Frequency Restoration Reserves bring the system back toward its target. European platforms such as PICASSO for aFRR and MARI for mFRR coordinate the exchange of balancing energy between TSOs.

Afterwards, imbalance-calculation and settlement systems compare the final measured position of each balance responsible party with its schedule. The physical deviation becomes a financial result.

This is the point where the two worlds finally meet. A forecast becomes a bid, a bid becomes a schedule, a schedule becomes a power flow, a power flow becomes a frequency deviation, and the deviation eventually becomes an invoice.

The grid and the market are not separate systems. They are two models of the same reality, connected by a large amount of software and by the stubborn requirement that, in every moment, production and consumption still have to match.

Regulation is part of the architecture

In most software projects, requirements begin with a customer, a product manager or somebody who has discovered a new button they urgently need.

In the grid, a requirement may have started years earlier in an EU regulation.

The chain roughly looks like this:

EU legislation → Network Codes and Guidelines → ACER and national regulators → national laws, tariffs and approved methodologies → company processes and technical requirements → software behaviour, interfaces and audit trails

For system operation, the System Operation Guideline, Regulation (EU) 2017/1485, defines rules around operational security, coordination, data exchange and planning between TSOs, DSOs and significant grid users.

For balancing, the Electricity Balancing Guideline, Regulation (EU) 2017/2195, establishes common principles for procuring, activating and settling reserves.

For day-ahead and intraday markets, the CACM framework defines how capacity calculation and market coupling should work across Europe.

Technical standards sit beside these legal layers. IEC 61850 shapes data models and communication inside utility automation. IEC 104 defines a common telecontrol path. Other standards define cybersecurity, grid-model exchange, metering, testing and many other details that developers eventually meet as requirements.

There is still room for architecture, design and creativity, but not every behaviour is free to invent. A configuration field may exist because a standard requires it. An audit trail may exist because an operator must later prove who issued a command. A market interface may have a strange deadline because an approved methodology defines a gate-closure time.

This was one of the most important discoveries for me. Regulation does not sit somewhere outside the software. It slowly flows down through organisations until it becomes a database field, a validation rule, a protocol message or a test case.

A map, not an encyclopedia

My goal was not to create the full catalogue of all software, protocols and standards used in the grid. That catalogue would be enormous and, more importantly, almost unreadable.

I wanted to draw a mental map that gives you an impression and a raw feeling of what exists, why it exists and where your own software takes its place.

The map has three large regions.

Operation and control software turns physical processes into measurements, alarms, models and commands.

Market software turns forecasts and electricity into bids, schedules, contracts and prices.

Coordination software stands between them and compares the planned world with the real one until they agree, or at least disagree within limits that the grid can survive.

Regulation and standards cover all three. They define many of the roads, borders and languages through which these systems communicate.

This is still not Google Maps. It is closer to Herodotus, with incomplete coastlines and unknown territory beyond the edge. But now, when I meet a new protocol, algorithm or application at work, I at least understand on which part of the continent I have landed.

In the future I want to visit some of these places properly. I would like to rebuild substation communication, follow a signal from a relay to a control centre, reconstruct a market-clearing algorithm, and understand how one planned megawatt becomes an imbalance and finally money.

The next topics will probably be defined by the problems I face in my current position and by questions from people who read this series. That feels like the most honest way to continue the journey.

Stay tuned.

And may the Force be with you.

All parts of the series:

Previous part:

I was new to the electricity sector as a software engineer. I came into the industry from web development.

The web was a pretty straightforward world. I usually knew for whom and why we developed a solution. Even when the software was highly technical, it was still clear who used it and who paid for it.

When I came into energy, everything was different.

Customers, users, regulators, and all possible three-letter abbreviations that meant companies, agencies, markets, services, and roles. It was hard to understand who actually bought the software. Who used it. Who paid for it. Who paid if something went wrong. Who defined how it should work. Who checked that it worked as it should.

And many... no. A LOT of other “who” and “why” questions.

Not knowing these answers might be acceptable in another industry. In energy, the structure of the sector shapes the architecture of the software. It defines requirements and explains why those requirements exist. A feature that looks strange from a purely technical perspective may be necessary because the customer must fulfil a legal, operational, market, security, or reporting obligation.

The person using the software may not be the person buying it. The company buying it may not be the organisation that defined the requirement. The asset owner may not be the system operator. The party that loses money when something fails may not be the party that approved the budget.

I am writing this article for colleagues who entered energy from other fields and are still trying to understand how the whole system fits together. By the end, you should have a clear mental model showing who owns the assets, who operates the power system, who buys and sells electricity, where the money comes from, and who defines the rules.

Subscribe now

Regulation created the structure

Laws, regulations, network codes, national rules, licences, and agreements define the main actors of the electricity sector and the way they interact. Therefore, regulation is not a separate layer added on top of an already existing market. It is one of the main forces that created the current structure.

The common EU framework is primarily established by the Electricity Directive, Directive (EU) 2019/944, and the Electricity Regulation, Regulation (EU) 2019/943. Together, they define common principles for generation, transmission, distribution, storage, supply, consumer participation, system operation, and electricity markets.

These documents are not implementation manuals for software engineers. They define something more fundamental: the environment in which energy companies, grid operators, markets, and eventually our software must work.

Why electricity is regulated so heavily

Electricity is a commercial product, but the power system is also critical infrastructure.

Electricity cannot be handled like a normal product that is manufactured, placed in a warehouse, and delivered later. Production and consumption must remain balanced continuously, while thousands of producers and millions of consumers use the same interconnected physical system.

The network itself also behaves differently from an ordinary competitive business. Building several competing transmission grids between the same regions would be enormously expensive and technically unnecessary. The same is generally true for local distribution networks. It would make little sense for several companies to dig up the same street and install parallel cables so that every household could choose its preferred physical grid.

Transmission and distribution networks are therefore treated as regulated monopolies. Competition is introduced where it is practical, particularly in generation, trading, and supply, while access to the physical networks is regulated.

This gives us the first part of the mental model:

Generation, trading, and supply can be competitive. Transmission and distribution are regulated network activities.

Because users cannot realistically choose another physical network, system operators cannot freely set any price or access condition they want. Their tariffs, investments, service obligations, and treatment of network users are supervised by regulators.

Why one company does not simply do everything

Historically, electricity systems were often organised around vertically integrated utilities. One company could generate electricity, own and operate the network, and sell electricity to consumers.

From a technical perspective, that structure looks convenient. One organisation owns the chain and coordinates it internally.

From a market perspective, it creates a conflict of interest.

Imagine a company that owns the transmission network and also owns power plants. Competing generators need access to the same network. The network owner could favour its own generation business by delaying connections, restricting access, using commercially sensitive information, or applying different conditions to competitors.

The EU response was unbundling, which separates network operation from potentially conflicting interests in generation and supply. The European Commission describes unbundling as one of the main pillars of the internal energy market because network operators must act independently and provide non-discriminatory access to infrastructure. The Commission’s overview of internal energy market governance explains the three main transmission unbundling models:

• Ownership unbundling, where network ownership and operation are separated from generation and supply interests.

• Independent system operator, or ISO, where the integrated company may retain formal ownership of the network, but an independent organisation operates, maintains, and develops it.

• Independent transmission operator, or ITO, where the network remains inside an integrated group but is operated through an independent subsidiary under detailed rules.

Distribution unbundling is less absolute. A distribution system operator that belongs to a vertically integrated group must generally be independent in its legal form, organisation, and decision-making, but EU law does not automatically require separate ownership of its assets. The directive also allows Member States to exempt some smaller integrated undertakings from parts of these requirements. The details are set out in Article 35 of the Electricity Directive.

The accurate conclusion is therefore not that a company can never participate in more than one part of the sector. The conclusion is that network operation must be sufficiently independent from competitive interests so that access remains fair.

Once this principle is introduced, the sector separates into several roles:

• generators produce electricity;

• transmission system operators operate high-voltage transmission systems;

• distribution system operators operate regional and local networks;

• suppliers sell electricity contracts to consumers;

• traders buy and sell electricity and related products;

• market operators organise trading;

• regulators supervise network and market activities.

These roles are not merely names invented by companies. They are part of the regulated design of the sector.

Who creates and enforces the rules

The regulatory chain begins at EU level, but not every rule comes from the same organisation.

The European Commission proposes legislation and develops energy policy. The European Parliament and the Council adopt EU legislation. Regulations apply directly, while directives define results that Member States must implement through national law.

The main legislation is supported by more detailed network codes and guidelines. These rules cover grid connection, system operation, emergency procedures, capacity allocation, congestion management, forward markets, day-ahead and intraday trading, and electricity balancing. ACER provides a useful overview of the binding European electricity market rules.

At national level, independent regulatory authorities supervise the market and regulated network businesses. Their responsibilities vary between countries, but they commonly include approving or reviewing network tariff methodologies, certifying system operators, supervising unbundling, monitoring competition, protecting consumers, and enforcing national and EU rules. In Austria, this role is performed by E-Control.

At European level, the European Union Agency for the Cooperation of Energy Regulators, or ACER, coordinates national regulators and acts where cooperation across borders is required. ACER contributes to common methodologies, monitors markets, and takes certain decisions when national regulators cannot reach agreement.

ENTSO-E and the DSO Entity have different roles. ENTSO-E coordinates transmission system operators and contributes technical expertise, common methodologies, planning, data publication, and operational cooperation. The DSO Entity performs a corresponding European coordination role for distribution system operators. Neither organisation is an independent public regulator.

Technical standards add another layer. Standards from organisations such as IEC, CENELEC, ETSI, and ISO may define communication protocols, security controls, equipment behaviour, interoperability, testing, and lifecycle requirements. A standard is not automatically a law, but it may become mandatory through regulation, a grid code, a licence, a contract, or a procurement specification.

The simplified chain looks like this:

Figure 1: A software requirement can travel a long way before it reaches a development team.

This explains why a small validation rule in Jira may originate from an EU regulation, be interpreted by a national regulator, translated into a process by a system operator, and finally become a requirement for a software vendor.

Who owns the assets of the power system?

There is no single European company that owns everything between a power plant and a wall socket. The power system consists of assets owned by generators, transmission companies, distribution companies, governments, municipalities, investment funds, industrial companies, cooperatives, and private households.

It helps to separate the physical system into four groups.

Generation assets

Generation assets convert another form of energy into electrical energy. They include nuclear, gas, coal, hydro, wind, solar, biomass, and other power plants.

Their ownership can be private, public, municipal, cooperative, or mixed. A large utility may own dozens of plants. An industrial company may own generation for its factory. A household may own rooftop solar panels.

Generation is generally a competitive activity. A generator earns revenue by selling energy and, depending on the technology and market, may also sell balancing capacity, balancing energy, ancillary services, certificates, or other products.

Transmission assets

The transmission grid transports large amounts of power over long distances at high voltage. It connects large generators, large consumers, distribution networks, and neighbouring transmission systems.

Transmission assets include lines, cables, substations, transformers, switchgear, protection systems, communication networks, and control centres. They are normally owned or controlled by certified transmission system operators (TSO), although the ownership structure differs by country and by unbundling model.

A TSO can be publicly owned, privately owned, publicly listed, or held through a mixed structure. The ownership model does not change the fundamental point: the company performs a regulated system function and must satisfy independence, access, operational, and certification requirements.

The ENTSO-E transmission system map gives a useful view of the major transmission infrastructure operated by European TSOs.

Distribution assets

Distribution networks connect most consumers and smaller generators to the power system. They contain regional substations, medium and low-voltage lines and cables, local transformers, switching equipment, meters, communication systems, and an increasing number of distributed resources.

Distribution system operators can be large national or regional companies, municipal utilities, private companies, public companies, or members of larger utility groups. Because distribution is a regulated monopoly, consumers usually cannot choose their DSO. Their physical location determines which network connects them.

They can, however, normally choose a supplier.

This distinction is fundamental:

The supplier manages the customer’s commercial electricity contract. The DSO operates the local network to which the customer is physically connected.

Changing supplier does not cause another company to install a second cable to the building. The commercial relationship changes, while the physical connection normally remains with the same DSO.

Customer and industrial assets

Not every electrical asset belongs to a generator or grid operator.

Factories, railway systems, data centres, mines, hospitals, airports, and large commercial sites may own substations, transformers, internal networks, generators, batteries, protection systems, and communication infrastructure. Households may own solar panels, batteries, heat pumps, and electric vehicle chargers.

This creates an important boundary between the public network and the customer installation. The grid operator defines connection conditions, but the customer owns and operates equipment on its side of the boundary. Protection, metering, remote control, data exchange, maintenance, and fault responsibilities must be clearly assigned.

Knowing who owns an asset is therefore only the beginning. We still need to ask who operates it, who controls it in real time, who defines its requirements, and who carries the consequences when it fails.

Who operates the power system?

Ownership answers who possesses the assets. Operation answers who is responsible for making the system work.

The word operator appears everywhere in the sector. A power plant has operators. A substation may be remotely operated from a control centre. A distribution network has a system operator. A transmission network has another system operator. A market also has an operator.

We therefore need to ask what exactly is being operated.

Transmission system operators

A transmission system operator, or TSO, is responsible for operating the transmission system and maintaining system security within its area of responsibility.

A TSO monitors the high-voltage network, coordinates outages, manages congestion, procures balancing services, plans network development, exchanges operational data with neighbouring TSOs, and prepares for emergencies and restoration. It is also responsible for maintaining the balance of the power system within the operational framework defined by European and national rules.

The unusual part is that a TSO is responsible for the behaviour of a system made mostly from assets owned by other organisations. It does not own every generator, industrial load, battery, wind farm, or distribution network connected to the system.

It operates through measurements, forecasts, schedules, market mechanisms, reserve procurement, control signals, grid topology, operating agreements, and emergency procedures.

Distribution system operators

A distribution system operator, or DSO, operates, maintains, and develops a regional or local distribution network.

A DSO connects consumers and distributed generators, manages planned and unplanned outages, restores supply after local faults, maintains voltage within permitted limits, provides connection information, and exchanges data with suppliers, generators, aggregators, consumers, and TSOs.

Historically, distribution systems were often treated as passive networks through which electricity flowed from the transmission grid toward consumers. That model is becoming less accurate as distribution networks contain more solar generation, batteries, electric vehicles, heat pumps, flexible industrial demand, and local energy communities.

A DSO therefore needs increasingly good observability, forecasting, automation, and coordination with the TSO.

Synchronous areas and operational responsibility

The European power system is physically interconnected, but operational responsibility still needs clear boundaries.

A synchronous area is a group of AC systems operating at the same nominal frequency and connected strongly enough that a significant imbalance affects frequency across the whole area. Europe contains several synchronous areas, with Continental Europe being the largest.

Inside a synchronous area, balancing and frequency-control responsibilities are divided into defined operational structures. The formal terminology includes load-frequency control blocks, load-frequency control areas, monitoring areas, and scheduling areas. These concepts are defined in the System Operation Guideline, Regulation (EU) 2017/1485.

For an introductory mental model, the important point is simpler:

The interconnected system is divided into defined operational areas so that schedules, measurements, deviations, reserves, and responsibility can be assigned to specific TSOs or coordinated groups of TSOs.

This is why we should not treat a country, TSO territory, synchronous area, control area, and bidding zone as synonyms. Sometimes their borders align. Often they solve different problems and therefore do not.

ENTSO-E and regional coordination

National transmission systems cannot be operated in isolation because power flows, outages, market schedules, and security problems cross borders.

ENTSO-E brings together 40 member TSOs from 36 countries to support coordinated operation, market integration, planning, common methodologies, data publication, and technical cooperation.

ENTSO-E is not a European super-TSO. It does not replace national TSOs and does not directly operate every substation. A TSO operates its transmission system. ENTSO-E coordinates the TSOs and fulfils tasks assigned to it by European legislation.

Regional coordination centres provide another layer of cooperation. They support TSOs with regional security analysis, capacity calculation, outage coordination, adequacy forecasting, and common grid models. The individual TSO remains responsible for its own system, while the regional centre provides calculations and a wider view that no national operator can produce alone. ENTSO-E describes these tasks in its overview of European power regions and regional coordination.

Who buys, sells, and pays?

When I started researching how the electricity market works, I expected it to be the most straightforward topic in this whole series.

The result was the opposite, and it was much more exciting than I expected.

The market opened another dimension of the power system for me. Its structure is almost as complex as the physical layer. The physical and commercial layers intersect and depend on each other, but they mostly exist as two parallel worlds with their own actors, rules, states, and processes.

In the physical world, we have generators, consumers, transmission lines, voltage, frequency, and power flows.

In the commercial world, we have contracts, bids, schedules, market positions, prices, and financial settlement.

Understanding the difference between these worlds is the foundation for understanding the electricity market.

The market is not the physical grid

My first question was simple: how does an electricity producer sell electricity and deliver it to a buyer?

The confusing answer is that it does not deliver its own electricity directly to that buyer.

Electrons in the network do not carry labels. We cannot take a particular group of electrons produced by Power Plant A and route it through selected transmission lines to Factory B. The physical power flow is determined by the grid topology, electrical impedances, voltages, and phase angles, not by the commercial contract.

A generator injects power into the interconnected system. A consumer withdraws power from the same system. Commercially, they may have bought and sold electricity between them, but physically there is no dedicated connection carrying only the electricity from that seller to that buyer.

This creates two parallel views of the same delivery period.

The market does not route electricity. It coordinates what participants promise to inject or withdraw and assigns financial responsibility when their actual behaviour differs from those promises.

Contracts create the commercial plan

A seller and a buyer agree on an amount of electrical energy for a particular delivery period and usually for a particular market area.

One participant promises to provide a commercial quantity. Another participant takes the corresponding commercial position.

Across the market, thousands of such trades create the expected commercial balance between production and consumption. Depending on the market and national rules, these positions are translated into schedules, nominations, and other information used by market participants and system operators.

However, the sum of all commercial positions does not guarantee that the physical grid will operate securely.

A market result may be commercially balanced while still creating too much power flow through a particular transmission corridor. A generator may fail after selling its expected production. Consumers may use more or less electricity than forecast. Wind and solar production may differ from predictions.

The TSO therefore uses market schedules together with forecasts, measurements, network models, equipment availability, and security analysis. Commercial planning tells the TSO what participants expect to do. Physical operation determines whether the system can actually support it.

Electricity is traded across different timeframes

Electricity is not traded through one single market.

Participants make decisions at different times because their information and risks change as delivery approaches. A supplier can estimate next year’s consumption, but tomorrow’s forecast will be more accurate. A wind producer knows more about expected generation a few hours before delivery than several months earlier.

The European wholesale market is therefore divided into several timeframes. ACER describes forward, day-ahead, intraday, and balancing as the main connected market stages. [^https://www.acer.europa.eu/electricity/market-rules/market-rules-different-electricity-market-timeframes]

Long-term and forward trading

Long-term contracts are made months or years before delivery.

They include bilateral contracts, power purchase agreements, forwards, futures, and other hedging instruments. Their main purpose is often not to predict every future megawatt-hour precisely, but to reduce exposure to future price changes.

A supplier may want predictable purchasing costs. A generator may need stable revenues to support an investment. A factory may want to protect itself from extreme electricity prices.

Day-ahead market

In the day-ahead market, participants buy and sell electricity for delivery during the following day.

They submit bids and offers for defined delivery periods. Market coupling combines these orders across bidding zones while accounting for the available cross-zonal transmission capacity.

The day-ahead result becomes an important commercial baseline, but it is still based on forecasts.

Intraday market

After the day-ahead market closes, reality continues to develop.

Weather forecasts change. A generator becomes unavailable. Consumption expectations move. A battery operator changes its planned position.

The intraday market allows participants to adjust their positions closer to physical delivery. This helps them reduce the difference between what they previously traded and what they now expect to produce or consume. European intraday trading continues close to real time, subject to the applicable gate-closure rules. [^https://www.acer.europa.eu/news/acer-greenlights-30-minute-intraday-electricity-gate-closure-time-across-eu-borders]

Balancing market

Even after intraday trading, actual production and consumption will not exactly match the commercial positions.

The TSO must correct the remaining physical imbalance in real time. It does this by activating balancing resources that can increase or decrease their generation or consumption.

The balancing market is therefore different from the earlier markets. Forward, day-ahead, and intraday trading allow participants to build and correct their own commercial positions. Balancing allows TSOs to procure and activate the physical flexibility needed to maintain system frequency and avoid a wider failure when the final result still does not match. [https://www.acer.europa.eu/electricity/market-rules/market-rules-different-electricity-market-timeframes]

Not every producer and consumer trades directly

It would be extremely difficult for every household, small generator, factory, and battery to buy and sell every delivery period independently.

The electricity market therefore contains actors that may not produce or consume electricity themselves but organise the commercial side of the system.

They aggregate customers, forecast consumption, access wholesale markets, manage schedules, trade changing positions, issue invoices, and carry financial responsibility.

Supplier

A supplier is the company that manages the commercial electricity relationship with the end customer.

For most households, this is the company that offers the tariff, signs the electricity contract, purchases energy for its customer portfolio, and sends the bill.

The supplier estimates how much electricity its customers will use and purchases corresponding quantities through contracts and markets. It also manages market access, forecasting, billing, customer service, and the financial consequences of differences between expected and actual consumption.

The supplier should not be confused with the DSO.

The supplier manages the customer’s commercial electricity contract. The DSO operates the local network to which the customer is physically connected.

Changing supplier normally changes who sells electricity to the customer. It does not change the physical cable or the local network operator.

What happens when the plan is wrong?

The grid must remain physically balanced, but no forecast is perfect.

A generator may produce less than planned because of a technical failure. A wind farm may produce more than forecast. Consumers may use more electricity than expected during a cold evening or less than expected during a holiday.

Suppose a supplier expects its customers to consume 50 MWh during one settlement period and purchases 50 MWh.

The customers actually consume 52 MWh.

Physically, the grid does not stop supplying them after the contracted 50 MWh has been consumed. The power system continues operating, and the TSO maintains the overall balance using the available physical response and balancing actions.

Commercially, however, somebody must be responsible for the missing 2 MWh and for the costs created by the deviation.

This is where the balance responsible party enters the picture.

Balance responsible party

A balance responsible party, or BRP, is a market role that accepts financial responsibility for the imbalance of a portfolio.

A BRP may manage a portfolio containing generators, suppliers, consumers, traders, storage systems, or aggregators. A company can become a BRP itself, or it can contract another BRP to carry its balance responsibility.

EU electricity-market rules require market participants either to be BRPs or to contractually delegate their balance responsibility to one, apart from limited exceptions. Each BRP is financially responsible for its imbalance and must strive to remain balanced or help the system remain balanced. [https://eur-lex.europa.eu/eli/reg/2019/943/oj/eng]

The BRP can be understood as the commercial boundary around a portfolio.

Inside that boundary, there may be many injections and withdrawals:

• power plants producing electricity;

• suppliers serving consumers;

• factories consuming electricity;

• batteries charging and discharging;

• traders buying and selling market positions.

The BRP tries to make the portfolio’s total commercial position match its expected physical behaviour.

What does a BRP actually do?

A BRP does not simply wait until an imbalance happens and then pay the bill.

It actively manages the portfolio before delivery.

Its work can include:

• forecasting production and consumption;

• collecting plans and availability information from portfolio members;

• purchasing or selling electricity in long-term and day-ahead markets;

• updating positions through intraday trading as forecasts change;

• submitting schedules and position information required by market rules;

• monitoring expected production and consumption;

• reacting to plant failures or forecast changes;

• receiving allocated metering data after delivery;

• calculating or checking the portfolio’s imbalance;

• settling the remaining imbalance with the connecting TSO.

Before the applicable intraday gate closure, a BRP can change schedules and use the intraday market to reduce an expected shortage or surplus. After delivery, the TSO calculates the difference between the BRP’s final position and its allocated physical volume for each imbalance settlement period. That difference is settled using the applicable imbalance price. The precise procedures vary between countries and balancing arrangements. [https://eur-lex.europa.eu/eli/reg/2017/2195/oj/eng]

Returning to our example, the supplier’s customers consumed 52 MWh while only 50 MWh had been purchased.

If the supplier belongs to a BRP portfolio, the shortage becomes part of the BRP’s total position. Other generators or consumers inside the same portfolio may partly offset it. If the portfolio still has a net shortage of 2 MWh, the remaining imbalance is settled between the BRP and the TSO.

The BRP therefore resembles an insurance provider only in a limited sense: it accepts responsibility for a risk that individual participants may not want to manage alone.

However, unlike normal insurance, it does not simply absorb the cost. It forecasts, trades, aggregates opposite deviations, charges its portfolio members, and manages the remaining financial exposure. Calling it a portfolio manager with balance responsibility is more accurate than calling it insurance.

For another practical explanation, the Next Kraftwerke overview of the BRP role provides a useful industry perspective.

BRP and BSP are not the same thing

The abbreviations are similar, but the roles solve different problems.

A BRP carries financial responsibility for the difference between the commercial position and the allocated physical behaviour of its portfolio.

A balancing service provider, or BSP, provides physical flexibility that the TSO can activate. A BSP may operate a power plant, battery, industrial load, aggregated portfolio, or another controllable resource capable of changing its injection or withdrawal.

One company may perform both roles, but they remain separate responsibilities.

Figure 2: The BRP manages the commercial deviation, while the BSP provides the physical response activated by the TSO.

Suppose a power plant unexpectedly disconnects.

Its BRP becomes exposed to the resulting commercial shortage. The BRP may have reduced that exposure through intraday trading before gate closure, or other assets in its portfolio may offset part of it.

At the same time, the TSO sees the physical power-system imbalance and activates balancing resources from BSPs.

The TSO solves the immediate physical problem.

The imbalance-settlement process assigns the commercial consequences.

This is where the two parallel worlds intersect.

The physical system cannot wait for contracts to be corrected. The TSO must keep it stable immediately. The commercial system then determines whose position differed from reality and who pays or receives money as a result.

That is the central purpose of the electricity market: not to route individual electrons from seller to buyer, but to coordinate planned injections and withdrawals, create prices, support efficient use of the network, and assign financial responsibility around one shared physical machine.

Putting the layers together

Consider a wind farm connected to a regional network and a factory consuming electricity elsewhere.

The physical flow begins with the wind farm injecting power into the interconnected system. The factory withdraws power through its local distribution network. The electricity reaching the factory is not a private stream routed from that particular wind farm.

The commercial flow is represented by contracts and market positions. The wind farm may sell through a trader or aggregator. A supplier buys electricity and sells a contract to the factory. A NEMO may operate the day-ahead or intraday market where part of the position is traded.

The operational flow is managed by system operators. The DSO operates the networks to which the wind farm and factory are connected. The TSO monitors the wider system, manages congestion, and procures balancing services. If wind production differs from the schedule, the physical system still has to remain balanced.

The financial-responsibility flow is assigned through balance responsibility. The relevant BRP carries the imbalance caused by the difference between scheduled and actual production or consumption. A BSP may provide balancing energy when activated by the TSO.

The regulatory flow begins with EU and national rules, continues through ACER and the national regulator, and becomes licences, tariffs, methodologies, grid codes, company processes, and technical requirements.

No single company owns or controls the complete process. The European electricity sector works through coordinated responsibility distributed among specialised actors.

What this means for software engineers

Once we understand the structure, many technical requirements stop looking arbitrary.

Suppose a team receives a requirement to record every control action with the user identity, timestamp, previous value, new value, source system, and reason for the change. From a narrow product perspective, this can look excessive.

However, the requirement may exist because several organisations share responsibility, an incident must be reconstructed, a regulator may require evidence, a control action can have safety consequences, or incorrect data can create financial liability.

The feature is not merely audit logging. It is the software representation of responsibility.

The same pattern appears in requirements for redundancy, strict time synchronisation, role-based access, configuration approval, long retention periods, traceable calculations, formal testing, standardised protocols, controlled updates, and operation during partial failures.

Some of these requirements come from physics. Some come from regulation. Most come from the interaction between technical reality and institutional responsibility.

When you begin work on an energy software system, ask eight questions:

1. Who owns the relevant asset?

2. Who operates the asset or process?

3. Who buys the software?

4. Who uses the software every day?

5. Who defined the requirement?

6. Who receives or relies on the data?

7. Who carries the operational and financial risk?

8. Who checks that the system complies with the rules?

The answers reveal the architecture around the architecture.

They explain why the system needs particular interfaces, security boundaries, access models, audit records, approval flows, availability targets, deployment procedures, and lifecycle guarantees.

Conclusion

When I entered the electricity sector, I tried to understand it as a simple chain. Someone produced electricity. Someone transported it. Someone sold it. Someone consumed it.

That picture was not completely wrong, but it was too small to be useful.

The sector is deliberately divided into specialised roles. Ownership, operation, commercial responsibility, and regulation are separate layers. A company may own an asset without operating the wider system. It may operate critical infrastructure while being privately owned. It may buy software whose requirements were defined by a regulator, a network code, or another operator.

Once these layers are separated, the collection of strange abbreviations starts to form a system.

More importantly, we can answer the questions that matter when building software:

Who is our customer? Who is our user? Who defined the requirement? Who pays for the system? Who carries the risk? And who will be responsible when something goes wrong?

These questions do not replace technical analysis. They tell us which technical problems actually matter.

All articles of the series:

Primary sources

• Directive (EU) 2019/944 on common rules for the internal market for electricity

• Regulation (EU) 2019/943 on the internal market for electricity

• European Commission: Governance of the internal energy market

• ACER: European electricity market rules

• ACER: Market rules for different electricity market timeframes

• ACER: Electricity balancing

• ENTSO-E member companies

• ENTSO-E regional coordination

• Commission Regulation (EU) 2017/1485 establishing a guideline on electricity transmission system operation

• European Commission: Electricity prices

Previous part:

Suppose you operate a web-service across three data centres. One region handles a large share of the traffic, and then it disappears.

Your monitoring detects failed health checks. The load balancer stops sending new requests to the failed region. Queues hold work that cannot be processed immediately. Clients retry some requests. Spare capacity in the remaining regions absorbs part of the load, autoscaling adds more instances, and operators receive alerts while the system settles into a degraded but usable state.

Software engineers know this story. We design distributed systems with observability, redundancy, failure detection, isolation, automated recovery and human escalation because components eventually fail.

Now replace the data centre with a 1,000 MW generator connected to the European power grid.

Subscribe now

The comparison still helps, but the failure is harder. Electricity cannot wait inside a queue until another generator is ready. Consumers continue withdrawing power at the same moment that generation disappears. The electrical state begins changing before a monitoring system has delivered the first alarm, and power redistributes through transmission lines according to physical laws rather than routes selected by an application.

The grid also has shared physical state. Frequency reflects the balance between active power generation and consumption. Voltage differs across the network and depends strongly on local reactive power and power flows. Current, equipment temperatures, rotor angles and line loading continue changing while software is still collecting measurements.

In Part 1, Electricity Is Not Water, we built the physical model behind generators, transformers, AC and frequency. In Part 2, The Grid Is Not a Pipeline, we assembled the network as a graph of substations, lines, generators, storage and loads. Now the machine is running, and we can ask how it stays alive when its state never stops changing.

In this post you will learn how protection, monitoring, operation and control work together, what SCADA contributes to that chain, which physical variables operators care about, and what happens from the first instant after a large generator disconnects until the grid reaches a new secure state.

Figure 1: A software platform can queue and redirect work. A power-system disturbance begins changing the shared physical state immediately.

A Distributed Physical System

The grid behaves like a distributed system in the broad sense. It spans many locations, has no single machine that contains its complete state, depends on communication and coordination, and must continue operating when individual components become unavailable.

The difference is that its nodes are coupled by physics.

A web service can often tolerate short disagreement between replicas. Messages can arrive late, requests can wait, and a control plane can calculate a new routing decision before the data plane changes. The electrical grid has far less freedom. A breaker opening in one substation immediately changes the topology seen by the electrical system. Voltages and currents respond at electromagnetic speed, rotating machines accelerate or decelerate, and power flows settle into new paths throughout the connected network.

This makes grid stability a broader idea than keeping the lights on. The system must remain within several limits at the same time.

Generation and load balance

A load is anything withdrawing electrical power from the grid. It may be one motor, a railway, a steel plant, a house or the combined demand of an entire distribution network.

Generators—and storage systems when discharging—inject active power, measured in watts. Active power is the average rate at which electrical energy is transferred into net work or heat: it turns motors, warms heaters, moves pumps and powers computers.

Across a synchronous AC system, generation and other active-power injections must continuously match consumption and network losses. When consumption and losses require more active power than generators and other resources provide, energy is initially drawn from the rotating masses of connected machines and frequency falls. When injections exceed consumption and losses, those machines accelerate and frequency rises.

In Continental Europe the target is 50 Hz. Frequency is therefore both a controlled quantity and a system-wide symptom. It tells operators that the active-power balance has changed, although it does not by itself reveal where the cause is located.

Voltage and reactive power

Every connection point in the grid also has a voltage. Voltage can be thought of as the electrical potential that drives current and allows power to move, but operationally the key point is simpler: voltage must stay within an acceptable range at every important bus and connection point.

Frequency is broadly shared across a synchronous area. Voltage is local. A bus in western Austria, a substation near Vienna and an interconnector in Germany can all have different voltage magnitudes at the same moment.

AC equipment needs changing electric and magnetic fields. Part of the current repeatedly stores energy in those fields and returns it later in the AC cycle. The associated quantity is reactive power, measured in var. It does not produce net work over a complete cycle, but it still creates current, occupies network capacity and affects voltage.

When an area lacks reactive-power support, voltage tends to fall. When it has too much, voltage can rise. The relationship is not perfectly isolated, because active power, reactive power, current, voltage and network impedance affect each other. Still, the following first model is useful:

Active power is closely connected to frequency, while reactive power is closely connected to voltage.

Power flow and transmission capacity

Power flow describes how active and reactive power move through the connected network. In software terms, the grid has a graph, but it has no router assigning each megawatt to a chosen path. Flows emerge from topology, line impedances, voltage magnitudes and phase angles.

If one transmission line opens, the power it carried does not wait for a new route. The electrical state changes and the flow redistributes over the lines that remain. A line far from the original event may receive much of the additional loading.

Each element has operational limits. A conductor can carry only so much current before heating becomes unacceptable. Transformers have thermal and equipment limits. Voltages must stay inside permitted ranges. The system also has stability limits that can become restrictive before a wire reaches its thermal rating.

This is what transmission capacity means operationally. It is not only a fixed number printed on a line specification. The usable capacity depends on the wider network state, including which elements are available, how power is injected and withdrawn, the weather, voltage conditions and what would happen if another component failed.

A grid can have enough total generation and still be unable to transport it safely to the loads that need it. This is why balancing generation and managing network congestion are related tasks, but they are not the same task.

Protection, Operation and Control

When I first tried to understand grid operations, these words blurred together. A relay trips a breaker, an operator opens another breaker, an automatic controller changes generator output, and SCADA displays all three events. It is easy to treat everything as one large control system.

The distinction becomes clearer when we ask what each area is trying to achieve.

Protection detects and isolates danger

A fault is an unintended electrical condition such as a short circuit, an insulation failure or an accidental connection between conductors and ground. Faults can create currents and voltages far outside normal operating values, damage expensive equipment and destabilise the surrounding network.

Protection systems exist to detect these conditions and isolate the affected equipment quickly.

A protection relay receives electrical measurements from instrument transformers or sensors. It applies configured logic to determine whether the measured condition indicates a fault or another dangerous state. Depending on the protected asset, the relay may compare currents entering and leaving a transformer, estimate the apparent distance to a fault on a line, watch for excessive current, or use measurements exchanged with a relay at the other end of the line.

When the relay decides that the equipment must be disconnected, it sends a trip signal to a circuit breaker. The breaker is the heavy electrical switch capable of interrupting normal load current and, within its rating, the much larger current produced by a fault.

The protected network is divided into protection zones around lines, transformers, busbars and generators. These zones usually overlap around breakers so that there is no unprotected gap. The design aims to remove the smallest practical part of the grid while still clearing the fault reliably. Backup protection covers cases where the primary relay, communication path or circuit breaker fails.

Protection is mainly local, automatic and fast. A line fault may need to be cleared within tens of milliseconds. The relay cannot wait for SCADA to send measurements to a control centre, for an operator to interpret an alarm and for a remote command to return. The trip path is designed to work without that sequence.

SCADA will report what happened. Operators will investigate and decide how to restore the network. Those steps matter, but they come after the immediate protection action.

Operation builds and maintains a secure state

Operation is the continuous work of understanding the current grid, preparing it for expected conditions and responding when reality differs from the plan.

Dispatchers in control rooms track the network topology, planned outages, unavailable equipment, generator schedules, weather, load forecasts, active and reactive reserves, voltages and line loading. They coordinate switching, release equipment for maintenance, return it to service, communicate with generators and distribution operators, and exchange information with neighbouring transmission system operators.

Routine work occupies much of this domain. Taking a transformer out of service for maintenance changes the grid before any failure occurs. A dispatcher needs to know whether the remaining network can carry the expected power and whether it can still survive another credible outage.

This leads to contingency analysis, which repeatedly asks questions such as:

• What happens if this line trips?

• What happens if the largest generator disconnects?

• Will another transformer overload?

• Will a bus voltage leave its permitted range?

• Which remedial actions would still be available?

The common N-1 idea means that the system should normally tolerate the loss of one relevant component without leaving acceptable operating limits after the allowed corrective actions. The exact application is more detailed than that sentence, but the mental model is valuable: operators manage the present while continuously testing possible next states.

EU system-operation rules require TSOs to monitor active and reactive flows, voltages, frequency, reserves, generation and load in real time, and to perform recurring contingency analysis against operational security limits. They also require close-to-real-time analysis to use state estimation. The complete rules are in the EU System Operation Guideline.

Control changes the state

Control is the set of actions that move the grid toward a desired operating state.

Some controls are built into equipment. A generator governor responds to frequency. An automatic voltage regulator changes excitation. A transformer tap changer adjusts its ratio. A capacitor controller switches a bank when voltage crosses a threshold.

Other controls are centralised. Automatic generation control sends changing active-power setpoints to participating resources. A control-room application can request a breaker operation. A dispatcher can instruct a power plant to change active or reactive output, order redispatch, or coordinate a topology change with another operator.

Operation and control therefore describe different parts of the same process. Operation provides the context, objectives and decisions. Control applies physical actions, either automatically or through commands initiated by operators. A protection relay also causes a physical action, but its purpose and logic are different: it removes danger immediately rather than optimising the wider operating state.

A useful software comparison is this:

• Protection resembles local safety logic with authority to shut down a dangerous component.

• Monitoring provides observability.

• Operation resembles the work of site reliability engineers who assess the complete service and choose a recovery strategy.

• Control contains the automated loops and commands that change the running system.

The comparison helps organise the responsibilities, although the grid remains coupled by physical laws throughout the whole process.

How the Grid Becomes Visible

A control room may have wall-sized diagrams and dozens of screens, but the operator still does not see the electrical system directly. The displayed grid is constructed from measurements, communication, equipment statuses and models.

The chain begins in the field.

Measurements begin at the equipment

Sensors and instrument transformers measure current and voltage. From these signals, devices calculate active power, reactive power, frequency and other electrical quantities. Breakers and disconnectors provide position indications. Transformers report tap positions, protection devices report trips, and equipment monitoring systems may provide temperatures, pressures and internal alarms.

A measurement always has context. A value needs a timestamp, a source, a unit and a quality indication. A current value from the wrong bay, a breaker state received late, or a voltage measurement marked good when its input circuit has failed can create a convincing but incorrect picture.

Substation devices collect and process data

Inside a substation, intelligent electronic devices, bay controllers, remote terminal units and substation automation systems collect measurements and statuses. Some values are processed locally. Some are stored as events. Some are forwarded to a remote control centre.

These devices may also receive commands. A control centre can request a breaker operation, change a setpoint or switch voltage-control equipment, subject to interlocks and operational procedures.

Protection lives close to this layer, but it should not be confused with telemetry. A protection relay can share measurements and event reports with the automation system while keeping its trip logic local and independent from the normal SCADA path.

Communication networks transport telemetry and commands

Substations and power plants are connected to control centres through operational communication networks. Depending on the system and its age, these links may use fibre, microwave, radio, leased communication services or other channels, often with redundant paths.

Protocols such as IEC 60870-5-104, IEC 61850, DNP3 and vendor-specific interfaces carry measurements, events, statuses and commands across different parts of the chain. The protocol tells us how information is represented and transported. It does not guarantee that the physical measurement is correct or that the model on the receiving side matches the real substation.

SCADA provides the supervisory interface

SCADA stands for Supervisory Control and Data Acquisition. In a grid control centre, it collects telemetry from many remote sites, presents measurements and equipment states, manages alarms and events, stores trends, and provides authorised operators with remote-control functions.

The word supervisory matters. SCADA gives a central view and allows high-level control of dispersed assets. It does not replace local protection, and it does not independently understand every physical consequence of an action.

A SCADA screen may show a one-line diagram in which a breaker appears open, a line is coloured according to its loading and an alarm list reports a generator trip. Behind that screen sits a long chain of measurement circuits, field devices, communication links, protocol gateways, databases, application logic and configuration.

NIST’s Guide to Operational Technology Security describes the same broad SCADA structure: field devices and RTUs or PLCs communicate through wide-area networks with control servers, operator interfaces, engineering workstations and historians in a control centre.

Figure 2: Monitoring information travels toward the control centre, while protection can trip equipment locally without waiting for a remote decision.

Why the screen can be wrong

SCADA is often described as real time, but that does not mean every point reflects physical reality at the same instant.

Measurements are sampled and transmitted at different rates. Communication links can fail. A value can become stale. A breaker indication can disagree with the actual contact position. A substation may be in a temporary configuration that the central model does not yet represent. Sensors have errors, and some values are calculated rather than directly measured.

This creates an observability problem. Operators need enough trustworthy information to determine the state of the network, even when individual measurements are missing or inconsistent.

State estimation reconstructs a coherent picture

State estimation combines telemetry with the electrical network model and the current topology. It checks whether measurements agree with each other, identifies likely bad data and estimates the bus voltages and phase angles that best explain the available information.

The resulting state is still an estimate. Its value comes from making the overall picture more coherent than a raw collection of telemetry points.

This estimated state becomes an input to other Energy Management System applications. Power-flow analysis calculates how power is moving through the network. Contingency analysis simulates possible outages. Security assessment identifies violations of operational limits. Forecasting tools compare the present with expected demand and generation.

Automatic generation control, usually implemented within the European load-frequency control process, uses frequency and control-area interchange information to calculate the area imbalance and adjust participating resources. It does not replace the dispatcher. It performs a continuous control loop inside objectives and limits established by grid operation.

SCADA is therefore a central part of the picture, but it is only one layer. The control room needs telemetry, models, specialised applications, automatic controllers, procedures and experienced operators. Treating the SCADA screen as the grid itself would be like treating a monitoring dashboard as the production system.

What the Grid Must Control

Once the operator has a usable picture of the network, the next question is what can be changed.

There is no single control knob for stability. Different mechanisms influence different parts of the operating state, and changing one variable often affects several others.

Frequency control and reserves

When active-power generation and consumption stop matching, frequency begins to move. The response is organised in layers because no resource is simultaneously instantaneous, unlimited, cheap and available everywhere.

The first physical response comes from stored energy in rotating machines and, increasingly, fast controls in batteries and inverter-based resources. This slows the frequency movement but does not remove the underlying imbalance.

The first organised reserve layer in Continental Europe is Frequency Containment Reserve, or FCR. Participating generators, storage systems and loads monitor frequency locally and change active power automatically. A falling frequency causes upward response, which means more generation or less consumption. A rising frequency causes downward response.

FCR is shared across the synchronous area. Resources respond to the common frequency deviation regardless of which control area caused it. Its immediate purpose is containment: stop the deviation from growing and stabilise frequency.

A reserve is available upward or downward power held by a prequalified resource. The resource might be a hydro plant with headroom, a thermal unit operating below maximum output, a battery with charge available, a pumped-storage plant that can stop pumping, or an industrial load able to reduce consumption. Reserving capacity means keeping part of that flexibility unused during normal operation so it is available when needed.

Once the common response has contained the disturbance, the responsible control area must restore its own balance. Automatic Frequency Restoration Reserve, or aFRR, is activated by the load-frequency controller, often described more generally as automatic generation control. It adjusts selected resources so frequency and scheduled cross-border exchanges move back toward their targets.

If the imbalance is too large or lasts too long, manual Frequency Restoration Reserve, or mFRR, provides another layer. It replaces part of the automatic response and prevents aFRR from remaining occupied indefinitely.

As a concrete Austrian example, Austrian Power Grid’s balancing overview describes FCR as an automatic response that must reach full activation within 30 seconds when the frequency deviation reaches 0.2 Hz. APG states that aFRR must activate within five minutes in the Austrian control area, while automatic and manual restoration reserves together must cover foreseeable balancing errors within 15 minutes. The exact products and rules belong to the European balancing framework and national implementation, but the sequence is the important part:

1. Slow the frequency change.

2. Contain the deviation.

3. Restore the responsible control area.

4. Replace the used reserves and prepare for another event.

Voltage control and reactive power

Frequency control works across the synchronous system. Voltage control is more local because voltage differs between buses and because transporting reactive power over long distances consumes current and network capacity.

Synchronous generators regulate voltage through their excitation systems. Increasing excitation allows a generator to supply more reactive power within its capability limits, while reducing excitation can make it absorb reactive power. Power-electronic converters can also provide controlled reactive current, although their capability depends on their design, current limits and active-power operating point.

Transformer tap changers adjust the turns ratio and therefore the voltage relationship between network levels. They are useful for maintaining a downstream voltage, but they do not create missing power. Under stressed conditions, repeatedly raising a distribution-side voltage can increase the demand seen by an already weak transmission system, which is why tap control may need to be coordinated or blocked during a voltage emergency.

Capacitor banks supply reactive power and tend to support local voltage. Reactors absorb reactive power and help reduce excessive voltage, especially on lightly loaded high-voltage lines and cables. Devices such as STATCOMs and synchronous condensers provide dynamic reactive-power support when voltage needs to change quickly.

The EU System Operation Guideline treats voltage and reactive-power management as a direct operational responsibility. It lists

transformer tap changes, capacitor and reactor switching, power-electronic devices, generator reactive-power setpoints and coordinated actions with distribution operators among the available remedial actions.

Figure 3: Operators manage several coupled problems at once. Active power strongly affects frequency, reactive power strongly affects voltage, and topology determines how both move through the network.

Power-flow control and redispatch

Operators cannot assign a route to each unit of power, but they can change the conditions that determine the flows.

Opening or closing selected breakers changes the topology. Transformer taps and phase-shifting transformers influence voltage and the distribution of active power. HVDC links can use controlled setpoints. Generation and consumption can be changed at specific locations.

One of the main tools is redispatch. A generator in one location increases output while another decreases output, keeping the total generation approximately balanced but changing where power enters the network.

Suppose a northern generator is increasing the loading of a constrained north-to-south corridor, while another plant closer to the southern load has spare capacity. Reducing the northern plant and increasing the southern plant can relieve the corridor without changing the total amount of generation.

Redispatch shows why a megawatt is not operationally identical in every location. The energy amount may be the same, but the effect on line loading, losses, voltage and contingency margins depends on where it is injected.

Normal Operation Is Already Active Work

A control room is not a fire brigade waiting for the next alarm. Normal operation requires continuous preparation because the safe state for the next hour may differ from the safe state now.

Demand follows daily and seasonal patterns but never matches a forecast exactly. Wind and solar output change with the weather. Generators start, stop and change output. International exchanges follow schedules while actual physical flows respond to the complete interconnected network. Equipment enters and leaves service for inspection, repair and construction.

Operators prepare switching programs before planned work. They confirm which breakers and disconnectors must change position, how the equipment will be isolated, what interlocks and safety procedures apply, and what the resulting topology means for the rest of the network.

An outage that is harmless during low demand may be unacceptable during a winter evening. Two individually reasonable maintenance plans may conflict because taking both assets out at once would remove the system’s ability to survive a further failure. This is why outage coordination extends across substations, companies and national borders.

The operator also watches operational limits. These include line and transformer loading, bus voltages, generator active and reactive capability, reserve availability, short-circuit levels and stability constraints. Some limits are immediate. Others permit a short overload while corrective action is prepared.

When analysis shows that one additional failure would violate a limit, the grid may enter an alert state even though all customers are still supplied and frequency is close to 50 Hz. The system is operating, but the safety margin has become too small.

The response might be preventive redispatch, returning an out-of-service line, changing topology, adjusting reactive-power resources, restricting a planned transfer or postponing maintenance. Good operation often looks uneventful because the risky state was corrected before a second event exposed it.

Coordination matters because the electrical system ignores organisational borders. A remedial action in one country can change flows in another. A distribution operator may control generators and reactive-power resources that affect the transmission grid. A power plant may own the machine, while the transmission operator is responsible for the wider system state. The EU System Operation Guideline formalises much of this data exchange and operational coordination between TSOs, DSOs and significant grid users.

A Large Generator Disconnects

We now have enough vocabulary to follow the failure from the opening example.

Assume a large synchronous generator is supplying 1,000 MW of active power and a meaningful amount of reactive power. A problem inside the plant causes its protection system to trip the unit and open the generator circuit breaker.

The exact numbers in the following sequence depend on the system state, but the order of responsibilities is stable.

The protection system disconnects the generator

The first action belongs to plant or generator protection. A relay detects a condition that makes continued connection unsafe, then commands the relevant circuit breaker to open.

This action protects the generator and the nearby electrical system. It may happen before the transmission control room knows that anything is wrong.

The breaker changes the topology. Active-power injection falls by 1,000 MW. The generator’s reactive-power contribution disappears as well. From that instant, the network has a new physical state.

Generation becomes lower than demand

Consumers have not coordinated with the failed plant. Their motors, heaters, trains and computers continue drawing power.

Total active-power generation is now lower than total active-power demand. The missing 1,000 MW is not supplied by one designated replacement generator during the first instant. The difference is initially supplied mainly by kinetic energy stored in the rotating masses of connected machines, while fast converter-based resources may also begin responding.

The electrical torque on the remaining synchronous generators becomes slightly greater than the mechanical torque supplied by their turbines. Their rotors slow down together, and the system frequency begins to fall.

This response is physical. No operator has selected it, and no SCADA command has caused it.

Local controls and FCR respond

Generator governors and other frequency-sensitive controllers detect the frequency change locally. Resources providing FCR begin increasing active-power injection or reducing consumption.

A hydro unit may open its guide vanes. A thermal unit may increase turbine input within its available headroom. A battery may inject power quickly. A pumped-storage plant may reduce pumping load. A flexible industrial consumer may lower demand.

These resources are distributed across the synchronous area. They do not need to know which generator failed. Frequency provides the common signal.

The reserve response grows while frequency is still falling. Eventually the additional power matches the deficit closely enough to stop the decline. The lowest point reached is called the frequency nadir.

If the response is sufficient, frequency stabilises. It may remain below 50 Hz because containment has stopped the movement without yet restoring the original balance.

Voltage and power flows change at the same time

The failed generator may have supported the voltage near its connection point. Losing its reactive-power capability can cause local voltage to fall, even while the frequency event is visible across the whole synchronous area.

Nearby automatic voltage regulators respond within their limits. Capacitors, reactors, converter controls or transformer taps may also act, depending on the network and configured control modes.

At the same time, active power begins entering the affected region from other generators through the transmission network. Flows redistribute across many lines. A corridor that previously carried moderate power may become heavily loaded because the local generator is no longer serving nearby demand.

This is where the graph from Part 2 becomes operational. The missing injection changes conditions across the network, and the most stressed line may be far from the failed plant.

Measurements and alarms reach the control centre

The generator breaker status changes. Active and reactive output measurements drop. Frequency values move. Nearby line flows and bus voltages change. Plant systems and substations create time-stamped events and alarms.

IEDs and RTUs forward this information through the operational communication network. SCADA updates the one-line diagram and alarm list. The operator may see a generator trip alarm, changed line colours, a frequency deviation and several secondary alarms caused by the same event.

These messages arrive after the physical system has already responded. They are evidence of the new state, not the trigger for the initial system-wide reaction.

Alarm handling is also harder than reading one clear error message. One event can produce hundreds of signals, some arriving out of order because different devices and communication paths have different delays. Operators and software need to distinguish the initiating event from its consequences.

State estimation and security analysis assess the new grid

The control centre now needs to answer several questions:

• Was the generator the only element lost?

• Are the received breaker states and measurements consistent?

• Which lines and transformers are now more heavily loaded?

• Has local voltage remained within its limits?

• How much FCR and reactive-power capability is still available?

• Can the grid survive the loss of another important component?

State estimation combines the new telemetry and topology with the network model. Power-flow and contingency-analysis applications calculate the current loading and simulate further outages.

The result may show that the frequency event is contained while the network is still unsafe. Perhaps one transmission corridor is close to its thermal limit. Perhaps local voltage is low. Perhaps the largest remaining generator has become the next critical contingency.

This is why returning frequency toward 50 Hz is necessary but insufficient. The full operating state must be secure.

aFRR and automatic generation control restore the area

FCR support came from across the synchronous area. The control area where the imbalance occurred must now restore its own balance and release that shared support.

The load-frequency controller calculates the control error from frequency and scheduled interchange. Through automatic generation control, it sends changing setpoints to resources providing aFRR.

These resources gradually add active power until the control-area imbalance is corrected. As aFRR takes over, FCR returns toward its neutral position and becomes available for the next disturbance.

If the outage lasts or the required correction is too large, mFRR and other slower actions replace part of the automatic response.

Operators create a new secure state

The failed generator may remain unavailable for hours. Operators cannot treat the restored frequency as the end of the incident.

They may redispatch generation so replacement power comes from locations that do not overload the network. They may start another plant, change pumped-storage operation, adjust cross-border schedules, switch reactive-power equipment, request new voltage setpoints or change the network topology.

They also need to restore reserve margins. A battery that delivered immediate power may need to recharge. Hydro and thermal units providing aFRR need room to respond in both directions again. The system should be able to survive another credible failure rather than remain balanced only while nothing else goes wrong.

The final state may look different from the state before the trip. A different set of generators is running, power flows use different corridors, reserve allocations have changed and some equipment may remain disconnected.

Recovery therefore means reaching a secure operating state, not recreating the exact previous state.

The same event on one timeline

• Milliseconds: Protection detects the plant condition and trips the breaker. Local and automatic protection.

• First electrical instant: Power flows change and rotating machines release stored energy. Physical response.

• First seconds: Frequency-sensitive controls and FCR increase power or reduce load. Distributed automatic control.

• Seconds to minutes: Telemetry, events and alarms reach SCADA; state estimation reconstructs the network. Monitoring and operational software.

• Minutes: aFRR and automatic generation control restore the control area. Central automatic control.

• Minutes to tens of minutes: mFRR, redispatch, voltage actions and topology changes establish a secure state. Automatic and operator-initiated control.

• Hours and beyond: Rescheduling, reserve replenishment, investigation and repair continue. Operation and asset management.

Figure 4: A generator loss is handled by several overlapping layers. Physics and local automation contain the first moments, while central controls and operators rebuild a secure system.

If the Normal Response Is Not Enough

The scenario above ends well because the protection system isolates the plant cleanly, reserves are sufficient and the network has enough remaining capacity.

A larger event, several simultaneous outages or a weak initial state can exhaust those layers.

If frequency continues falling, automatic under-frequency schemes can disconnect blocks of load. Removing consumers is damaging, but it can rebalance an island or synchronous area before generators also disconnect and the entire system collapses.

If voltage keeps falling, operators and automatic schemes may use all available reactive-power support, block transformer tap changers, reduce voltage targets or disconnect demand. If power flows remain outside safe limits, equipment may be switched, generation may be redispatched, transfers may be reduced and, in an emergency, selected loads may be disconnected.

These measures are defined in system defence plans. The European framework for emergency and restoration operation is set out in Regulation (EU) 2017/2196.

Protection still has a difficult role. If an overloaded line trips, its flow redistributes over the remaining network. Another line may then exceed its limit and trip as well. Each relay may be correctly protecting its own asset while the sequence weakens the wider system.

This is the basic mechanism behind a cascading outage:

1. One element disconnects.

2. Flows and voltages redistribute.

3. Another element exceeds its limits.

4. Protection disconnects it.

5. The remaining network receives even more stress.

The grid may eventually split into electrical islands. Each island must immediately balance its own generation and load. An island with too little generation experiences falling frequency, while an island with too much generation experiences rising frequency.

Emergency controls attempt to stop the cascade, preserve stable islands and keep as much of the system supplied as possible.

Different Disturbances Stress Different Layers

The generator loss gives us a useful complete story, but it should not become a template for every event.

A transmission line fault begins with a short circuit or another abnormal electrical condition. Line protection detects the fault and opens breakers at one or both ends, often within a fraction of a second. Generation and demand may remain nearly balanced, so the initial frequency change can be small, while the main operational problem is redistributed power flow and reduced transmission margin.

A transformer outage can remove the connection between voltage levels or parts of a substation. The result may be a local supply problem, overloaded parallel transformers, changed short-circuit levels or a voltage issue rather than a large system-wide frequency event.

A sudden rise in reactive-power demand, perhaps from large motors or a heavily loaded region, can depress local voltage while total active-power generation still matches consumption. Frequency can remain close to 50 Hz while part of the network approaches voltage instability.

A regional separation creates the clearest example of the system’s coupled nature. Once tie lines open, the two sides no longer share one active-power balance. Each island follows its own imbalance and frequency trajectory.

Protection, monitoring, operation and control appear in every case, but their order and relative importance change with the physical disturbance.

Where the Distributed-Systems Analogy Helps

We can now return to the failed data centre.

Both systems need observability. Operators need trustworthy measurements, event histories, health information and models that explain what is connected and available.

Both need spare capacity. A service without compute headroom cannot absorb traffic from a failed region. A grid without active-power reserves, reactive capability or transmission margin cannot absorb an electrical outage safely.

Both use layered response. Local components handle some failures automatically, central systems coordinate wider recovery, and human operators intervene when the event crosses the assumptions built into automation.

Both can operate in a degraded state. A service may remain available with higher latency and reduced redundancy. A grid may restore frequency while running with tighter transmission margins and less reserve.

Both need to prepare for the next failure. Recovering the current workload is not enough if the platform has lost all remaining redundancy. Restoring generation-load balance is not enough if the grid can no longer survive another contingency.

These similarities make distributed systems a useful starting point for developers entering the energy industry.

Where the Analogy Stops

The limitations matter just as much.

Software work can often be queued. Electrical load continues interacting with the network at every instant.

Software traffic can be assigned to a region or route. AC power divides across the connected network according to impedance, voltage and phase angle.

Many software failures remain local until another component calls the failed service. An electrical disconnection changes the physical state of the wider network immediately.

A monitoring system in software may detect a problem before users notice it. In the grid, frequency, voltage and current may already be moving before the control centre receives the first event.

Software replicas can disagree temporarily while eventual consistency resolves the difference. The synchronous grid has shared physical variables that must remain inside narrow limits continuously.

Finally, grid control actions feed back into the same physical system being measured. Opening a breaker to remove an overload changes flows elsewhere. Raising a downstream voltage can increase demand on an upstream network. Increasing one generator can relieve frequency while worsening congestion.

This is why grid software cannot be designed only as ordinary enterprise software with stricter availability requirements. It participates in a control loop around a large physical machine.

The Mental Model to Keep

A developer does not need to become a protection engineer or a control-room dispatcher to understand where grid software fits. The following model is enough to organise the vocabulary:

1. Physics moves first. Changes in generation, load or topology immediately affect frequency, voltage, current and power flow.

2. Protection removes immediate danger. Relays and circuit breakers isolate faults locally and quickly.

3. Monitoring constructs visibility. Field measurements, substation devices, communications, SCADA and state estimation create an operational picture.

4. Control changes the system. Governors, voltage regulators, reserves, automatic generation control, tap changers, reactive-power devices, redispatch and switching move the grid toward acceptable conditions.

5. Operation chooses and coordinates the secure state. Dispatchers assess limits, contingencies, outages and available actions across organisational and national boundaries.

None of these layers works alone. Protection can clear a fault but cannot replace lost generation. SCADA can show an overload but cannot decide which market and operational constraints matter. Reserves can restore active-power balance but cannot help if the replacement power has no safe transmission path. Operators cannot act effectively when measurements or models are wrong.

The European grid stays stable because these layers overlap across different timescales. Local physics and automation handle the first instant. Wide-area controls restore balance. Software helps operators understand the changing state. Human decisions rebuild enough margin for the next event.

In Part 1, we assembled the electromagnetic machine. In Part 2, we assembled the graph. Here, the graph became operational.

It is always changing, always measured and always being corrected. Stability is the result of that continuous work.

Next part:

All parts of the series:

References

• European Commission, Regulation (EU) 2017/1485 establishing a guideline on electricity transmission system operation

• European Commission, Regulation (EU) 2017/2196 establishing a network code on electricity emergency and restoration

• Austrian Power Grid, Balancing and frequency reserves

• NIST SP 800-82 Rev. 3, Guide to Operational Technology Security

This article connects two earlier series. In the web-server series, I rebuilt the HTTP side from raw sockets up to routing and file serving. In the TLS series, I rebuilt the secure-channel side: key exchange, certificates, key derivation, and encrypted records.

This piece is where the two meet: what has to change when a web server stops receiving plaintext HTTP bytes and starts receiving encrypted TCP bytes?

Subscribe

It did not.

The HTTP parser stayed boring. The router stayed boring. The file-serving code stayed boring.

The real change happened one layer below HTTP.

I had two separate learning projects before this:

• a small web server built from scratch: sockets, HTTP parsing, location matching, file serving, and eventually a single-threaded event-loop server;

• a small TLS-like secure channel built from scratch: ephemeral key exchange, certificate chains, server authentication, HKDF, AES-GCM records, and the difference between identity keys and session keys.

Putting them together led to one question:

What changes in the web server when the bytes coming from TCP are encrypted?

The useful answer is smaller than I expected:

TCP socket
   ↓
TLS handshake / record layer
   ↓ decrypted plaintext bytes
HTTP parser / router / file serving
   ↓ plaintext HTTP response
TLS record protection
   ↓ encrypted bytes
TCP socket

That is the whole architecture of the companion project.

This is not production HTTPS. It does not implement the real TLS 1.3 wire format, and browsers will not connect to it with https://localhost:8443.

It is a learning project. The goal is to make the integration boundary visible.

Once that boundary becomes clear, HTTPS feels much less magical.

The plain web server had one assumption

Before TLS, the web server had a simple mental model.

A client connects over TCP. The server reads bytes from the socket. Those bytes are HTTP bytes. The server appends them to a per-connection buffer, tries to parse an HTTP request, matches the path, reads a file, and writes an HTTP response.

The flow looked like this:

accept TCP connection
   ↓
read bytes from socket
   ↓
append bytes to HTTP input buffer
   ↓
parse HTTP request
   ↓
match URL against location config
   ↓
read file from root
   ↓
write HTTP response

The buffer is important because TCP is a stream.

One recv() call can give you half a request. Or exactly one request. Or multiple requests joined together. The parser cannot assume that one socket read equals one HTTP request.

In my server, the parser consumes complete HTTP messages from a buffer. If the request is incomplete, the server waits for more bytes. If there are extra bytes after a complete request, they stay in the buffer for the next parse.

For plain HTTP, socket bytes and HTTP bytes are the same thing:

context.http_input.append(data)
self._handle_http_messages(context)

That line works only because the bytes from the socket are already plaintext HTTP.

TLS breaks that assumption.

After TLS, socket bytes are no longer HTTP bytes

Once TLS is added, the TCP socket no longer carries a readable HTTP request.

The client still wants to send this:

GET / HTTP/1.1
Host: localhost
Connection: close

But that request is not placed directly on the wire.

It is wrapped inside TLS records. After the handshake, those records are encrypted and authenticated. The bytes arriving at the server socket are now protocol bytes for the TLS layer, not text for the HTTP parser.

That is the shift:

plain listener:
TCP bytes are HTTP bytes

TLS listener:
TCP bytes are TLS records
TLS records decrypt into HTTP bytes

If the HTTP parser starts caring about certificates, keys, record sequence numbers, or AES-GCM tags, the abstraction has leaked.

The HTTP parser should parse HTTP.

The TLS layer should turn unsafe network bytes into authenticated plaintext bytes.

The integration point belongs below HTTP

Production servers use the same high-level boundary.

NGINX does not ask the HTTP parser to understand encrypted TLS records. A TLS implementation such as OpenSSL handles the connection security first. After that, the HTTP processing layer receives plaintext HTTP bytes.

My project copies that architecture in a smaller educational form:

Production-ish mental model:
TCP → OpenSSL/TLS state → plaintext HTTP → NGINX HTTP processing

This project:
TCP → ToyTLSServerConnection → plaintext HTTP → HTTPParser / route matcher

The analogy is architectural, not protocol-compatible.

The project uses a small TLS-like protocol so the moving pieces are possible to read in one sitting. The handshake messages are simplified. The record framing is simplified. Many real TLS 1.3 details are missing on purpose.

But the boundary is the point:

socket bytes in
   ↓
connection layer decides whether they are plain HTTP or TLS records
   ↓
HTTP layer receives plaintext HTTP bytes either way

That boundary lets the same HTTP code serve both ports.

The config makes the boundary visible

The server uses an NGINX-style config because that was part of the original web-server project.

A single server block can listen on a plain port and on a TLS port:

http {
    server {
        listen 8080;
        listen 8443 ssl;
        server_name localhost;

        ssl_certificate certs/server_cert.pem;
        ssl_certificate_key certs/server_key.pem;
        ssl_certificate_chain certs/intermediate_cert.pem;

        location / {
            root html;
        }
    }
}

listen 8080; creates a normal HTTP listener.

listen 8443 ssl; creates a listener that attaches TLS state to every accepted connection.

Both listeners still share the same routing and file-serving code. The difference is what happens between recv() and HTTPParser.parse_message(...).

That is the part I wanted the project to expose.

The blocking TLS demo had to become a state machine

The original TLS demo was linear and blocking.

That is perfect for teaching the handshake:

receive ClientHello
send ServerHello
send ServerAuth
receive encrypted request
send encrypted response

But the web server I wanted to integrate with was not a one-client-at-a-time demo. It was a selectors-based server.

One event loop handles listening sockets and client sockets. A slow client should not freeze the whole process. That means the TLS implementation cannot sit inside a function waiting for the next record to arrive.

So the TLS code had to become buffer-oriented.

Instead of blocking on the socket, the TLS connection exposes three operations:

tls.feed_wire_data(socket_bytes)
pending = tls.pop_pending_wire_data()
plaintext = tls.read_plaintext()

feed_wire_data(...) accepts whatever bytes TCP happened to deliver.

pop_pending_wire_data() returns handshake bytes or encrypted application data that need to be sent back to the client.

read_plaintext() returns decrypted HTTP bytes after the TLS layer has enough complete records to process.

The web server does not need to know whether TCP split a TLS record into three reads or merged several records together. The TLS record buffer owns that problem.

That small interface is the bridge between the two projects.

Plain and TLS reads now differ by one layer

Here is the core read path in the integrated server.

For plain HTTP, the socket bytes go directly into the HTTP input buffer:

# Plain path: socket bytes are already HTTP bytes.
context.http_input.append(data)
self._handle_http_messages(context)

For the TLS listener, the same socket bytes first pass through the TLS connection state:

# TLS path: socket bytes are handshake/application records, not HTTP.
context.tls.feed_wire_data(data)
self._queue_output(context, context.tls.pop_pending_wire_data())

plaintext = context.tls.read_plaintext()
if plaintext:
    context.http_input.append(plaintext)
    self._handle_http_messages(context)

The HTTP handler never receives encrypted bytes. It receives plaintext after the TLS layer has done its job.

After that, request handling is the same:

request, consumed = HTTPParser.parse_message(context.http_input.data)
context.http_input.reduce_data(consumed)
response, should_close = build_http_response(request, context.server_block, self.base_dir)
self._send_application_bytes(context, response)

The response path mirrors the read path.

The HTTP layer builds a normal plaintext HTTP response. If the connection is plain HTTP, those bytes are written directly to the socket. If the connection is TLS, the server wraps the response in an encrypted application record first.

if context.tls is not None:
    wire = context.tls.protect_application_data(plaintext_response)
else:
    wire = plaintext_response

This is the part I like most about the final design.

The web server still feels like a web server.

TLS becomes a connection concern, not an HTTP concern.

What the toy TLS layer does

The TLS-like layer in this project is intentionally small, but it keeps the important shape of the original TLS series.

The handshake starts with ephemeral X25519 keys.

The client sends a ClientHello containing its ephemeral public key. The server generates its own ephemeral key for this connection and sends back a ServerHello.

Then the server sends authentication material:

• the server certificate,

• intermediate certificates,

• a signature over the two ephemeral public keys.

That signature is the simplified version of CertificateVerify. It proves that the peer presenting the certificate also controls the matching private key for this specific key exchange.

After that, both sides compute the same X25519 shared secret and derive directional AES-GCM keys through HKDF.

Directional keys matter. The client-to-server key protects client records. The server-to-client key protects server records.

client_write_key: records sent by the client
server_write_key: records sent by the server

Application data records also carry sequence numbers into the authenticated encryption. That lets the receiver detect tampering and sequence mismatches.

Again, this is not real TLS 1.3. The real protocol has much more structure: transcript binding, alerts, Finished messages, SNI, ALPN, session resumption, real record headers, and many other details.

For learning the web-server boundary, the smaller version is enough:

handshake creates keys
records protect HTTP bytes
HTTP parser sees only decrypted bytes

The useful lesson was not only cryptography

I started the project thinking the interesting part would be TLS.

The cryptographic pieces do matter. It is useful to understand why ephemeral keys exist, why certificates are about identity rather than encryption, and why authenticated encryption needs sequence numbers.

But the integration lesson was more general.

When you add a lower-level protocol to an existing system, the hard part is often choosing the seam.

If the seam is wrong, everything above it starts learning details it should not know.

If the seam is right, most of the application code stays boring.

In this case, the seam is between TCP and HTTP.

The HTTP parser should not know about X25519. The route matcher should not know about certificates. The file-serving code should not care whether the client used the plain port or the TLS port.

Those parts operate on HTTP.

The connection layer is responsible for turning the outside world into HTTP bytes.

That pattern shows up in a lot of infrastructure code:

• network bytes become parsed messages,

• encrypted records become plaintext streams,

• unreliable external systems become retried operations,

• hardware signals become typed events,

• logs become structured facts.

A design question I keep coming back to is:

What should the next layer be allowed to assume?

For this project, the HTTP layer is allowed to assume one thing:

I receive plaintext HTTP bytes.

Everything below that belongs to the connection layer.

Why build this if it is not real HTTPS?

A fair objection is: if the project is not browser-compatible, why build it?

For me, browser-compatible TLS is too large if the goal is to understand the integration boundary.

If I used OpenSSL directly, the project would be more practical, but the interesting internals would disappear behind a library call.

If I tried to implement full TLS 1.3, the project would become mostly about protocol correctness. That is valuable, but it would bury the web-server lesson under too much detail.

The educational middle ground is a TLS-like secure channel with the same high-level shape:

handshake
certificate verification
key derivation
encrypted records
application data

Then I can integrate that channel into the web server as if it were a real TLS stack.

The result is small enough to read, but realistic enough to expose the architecture.

That is the kind of learning project I like: simplified around one specific question.

Here the question is:

What does a web server need from TLS?

My answer after building it:

A secure byte stream that produces plaintext HTTP for the existing HTTP layer.

How to run the companion project

The repository is here:

https://github.com/DmytroHuzz/tls_web_server

The visual walkthrough is here:

https://dmytrohuzz.github.io/tls_web_server/

From a fresh clone:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e ".[dev]"

Then run the one-command demo:

python3 scripts/demo.py

It starts the server in a background thread, sends one plain HTTP request, then sends one HTTP request through the self-written TLS-like layer.

Both paths should return:

HTTP/1.1 200 OK

There are also separate server_side.py and client_side.py files if you want to see the workflow from each side, closer to the style of the original projects.

The tests cover the integration points that are easy to get wrong:

• fragmented TLS handshake records,

• fragmented encrypted application records,

• wrong DNS name certificate rejection,

• tampered encrypted record rejection,

• sequence-number mismatch rejection,

• HTTP requests split across multiple encrypted TLS records,

• the same web server serving plain HTTP and HTTP-over-self-written-TLS.

The mental model I keep now

Before building this, I would have described HTTPS as “HTTP with encryption.”

That is fine as a user-level description, but it is not precise enough when writing the server.

The server-side model that helped me is this:

TCP gives bytes.
TLS turns encrypted bytes into authenticated plaintext bytes.
HTTP parses the plaintext bytes.

That sounds simple, but it changes where the code belongs.

You do not sprinkle TLS checks through the HTTP parser.

You do not make the route matcher aware of certificates.

You do not duplicate the file server for secure and insecure connections.

You put a protocol layer below HTTP and keep the rest of the server honest.

That is what I wanted from this project.

Not a production HTTPS server. Not a replacement for OpenSSL.

Just a clearer picture of where HTTPS belongs.

If you want to follow the full path from encrypted TCP bytes to a parsed HTTP request and back again, I put the code and visual walkthrough here:

• Companion repo: https://github.com/DmytroHuzz/tls_web_server

• Visual walkthrough: https://dmytrohuzz.github.io/tls_web_server

Figure 1: The old picture says power flows through a pipe. The real grid is a synchronized graph that must stay balanced in real time.

In the first part, we killed one comfortable idea: electricity is not water.

Now we can kill the next one.

The power grid is often imagined as a large water system. Somewhere far away there is a power plant, like a pump. Long power lines carry electricity, like pipes. The wires reach a city, then a street, then a house, and when someone opens the “tap” by switching on a device, electricity flows from the power plant into the socket.

It is simple. It is visual. It feels intuitive.

And it is almost the worst possible picture if you want to understand the real grid.

The grid is not a pipe where energy sits inside wires waiting to be consumed by devices. A power plant does not fill transmission lines with electrons and send them to your laptop, fridge, or washing machine. The real grid is a synchronized electromagnetic system. It works in real time. It oscillates. It has a frequency. It connects generators, consumers, transformers, substations, transmission lines, distribution grids, storage systems, and control centers into one huge machine.

And this machine has one brutal rule: generation and consumption must stay balanced all the time.

Consumers constantly change their behavior. Someone starts an industrial motor. A train accelerates. A factory begins a shift. A city wakes up in the morning. A cloud moves over a solar park. Wind generation rises or drops. A power plant ramps up, ramps down, or disconnects. Nothing is static, and still the whole system must remain synchronized.

In continental Europe, this synchronization is visible through the 50 Hz grid frequency. APG describes grid frequency as the measure of balance between electricity supply and demand, and one of its core tasks is to keep the system close to 50 Hz continuously. Austrian Power Grid

So no, the grid is not a pipe.

It is closer to a living technical machine whose operating state changes all the time. It is measured, switched, protected, transformed, corrected, and balanced continuously. Not because engineers enjoy complexity, but because the system would not work otherwise.

In this post we will discuss a better model for understanding the grid. We will stop looking at the fake “power plant to socket” diagram and rebuild the picture from the center outward. We will start with the high-voltage transmission grid, then attach substations, generators, storage systems, distribution grids, and consumers around it. After that, we will inspect a real slice of the Austrian grid and see how these abstract ideas appear in real infrastructure.

Because once we stop seeing the grid as a pipe, we can finally start seeing it as a graph.

The old diagram is already misleading

Most explanations of the grid start with a power plant.

The diagram usually looks like this:

Power plant. Transmission line. Substation. Distribution line. House. Socket.

Figure 2: This picture is easy to understand, but it already teaches the wrong architecture.

At first sight, this seems reasonable because electricity is generated somewhere and consumed somewhere else. But the moment we draw the system this way, we accidentally teach the reader the wrong architecture.

We make it look like electricity has a source, a route, and a destination. We make it look like a delivery system. A factory produces a product, a road transports it, and a customer receives it.

But the electrical grid is not a delivery route.

A generator does not produce a private stream of electricity for one city. A house does not receive electrons that started their journey at one specific power plant. A transmission line is not a pipe with a fixed direction. The grid is a shared synchronized system. Generators inject power into this system. Consumers withdraw power from it. Storage can do both. Power flows change depending on the current physical state of the network.

That is why starting with a power plant is already dangerous. It keeps us inside the pipe model.

A better starting point is the network itself. Not the generator, not the socket, but the grid.

The grid is a graph, not a chain

For a software developer, the better mental model is not a pipe. It is a graph.

Figure 3: A better model: high-voltage lines form the backbone, substations are nodes, and generation, storage, distribution, and consumers attach to the graph.

Transmission lines are edges. Substations are nodes. Generators, storage systems, distribution grids, industrial consumers, cities, and interconnectors are attached to this graph at different points.

But we need to be careful here, because this analogy can also become stupid if we push it too far.

The grid is not the internet. Power does not move like packets. There is no router that says, “send this megawatt through line A and this other megawatt through line B.” Power flows according to physics. It depends on voltage, impedance, phase angle, topology, and the current state of the synchronized system.

Still, the graph model is extremely useful because it breaks the one-way chain illusion.

In a chain, you start at one end and walk to the other end. In a graph, you first ask what is connected to what. That is exactly how we should look at the grid.

The high-voltage network is the backbone. Substations connect parts of the backbone to each other and to lower voltage levels. Generators inject power at some nodes. Consumers withdraw power through other nodes. Storage systems can either withdraw or inject, depending on the operating mode. Interconnectors connect countries and control areas. Protection systems watch the graph and disconnect faulty parts when something goes wrong.

This is already a much better picture.

The grid is not a road from a power plant to your socket. It is a synchronized graph whose state must be kept inside safe limits.

Why we start with the high-voltage grid

If we start with a house, the grid looks like a local supply problem.

If we start with a power plant, the grid looks like a delivery problem.

But if we start with the high-voltage grid, the architecture becomes visible.

The high-voltage grid is the strong skeleton of the system. It connects regions, countries, large generation sites, large consumption areas, and lower-voltage networks. In Austria, APG manages the transmission grid at 110 kV, 220 kV, and 380 kV. These voltage levels are used to transport electricity efficiently over long distances and to connect large parts of the country into one operating system. Austrian Power Grid

This is where the pipe analogy starts to completely fall apart.

A high-voltage line is not just a long cable from one plant to one city. It is part of a larger network. Depending on generation, consumption, outages, maintenance, weather, and market schedules, power flows can change. A line can carry more or less power. A region can import or export. A storage plant can consume power in pumping mode and produce power in turbine mode. The same physical infrastructure participates in many different operating situations.

At the European scale, this becomes even more obvious. ENTSO-E (European Network of Transmission System Operators for Electricity) provides a transmission system map showing power plants, converters, substations, and high-voltage cables and lines operated by European transmission system operators. The map is useful for seeing the scale and density of the interconnected transmission network, but it should not be read as exact geography because ENTSO-E notes that network elements are not placed at their precise geographic positions. ENTSO-E Transmission System Map

Figure 4: Even as a schematic, the European transmission system looks less like a pipe and more like a continental graph. Source: ENTSO-E.

That map is overwhelming, and that is exactly the point.

The European grid is not a simple line from source to load. It is a continental machine. So instead of trying to understand the whole map at once, we can zoom into one country and then into one real grid node.

Austria is a good place to do this because the APG grid gives us a concrete high-voltage backbone, and the Austrian grid does not stop at the border. It is part of the wider European synchronous system.

Figure 5: Austria’s transmission grid gives us a real backbone to inspect, with 110 kV, 220 kV, and 380 kV layers connected through substations. Source: Austrian Power Grid.

Substations are the nodes where the graph becomes real

Once we think in graphs, substations become much more interesting.

A substation is not just “the place where voltage goes down.” That is the school version. It is not completely wrong, but it is far too small.

A substation is a grid node.

It is a place where lines meet, voltage levels connect, transformers change voltage, breakers can connect or disconnect parts of the system, measurements are collected, faults are detected, and operators can change the topology of the network.

In graph language, a substation is where the abstract edge-and-node model becomes physical.

A transmission line arrives. Another line leaves. A transformer connects the 380 kV grid to the 220 kV grid or the 110 kV grid. A distribution network may be supplied from there. A power plant may be connected nearby. A breaker can disconnect a faulty line. Protection systems can isolate a damaged part of the network before the fault spreads further.

That is why substations are not secondary details. They are one of the main reasons the grid can be operated at all.

But in this article, we will not yet open the substation and inspect every component. Busbars, bays, breakers, disconnectors, transformers, protection relays, SCADA, and IEC 61850 deserve their own article. For now, we need only one idea:

A substation is a controllable node of the grid graph.

A real example: Pongau substation

Now let us stop speaking only in abstractions.

APG’s (Austrian Power Grid) Pongau substation is a good real-world example because it shows how much can happen at one node of the grid.

Pongau is a region in the Austrian federal state of Salzburg, south of the city of Salzburg and close to the Alpine hydropower and pumped-storage area around Kaprun and Tauern. You do not need to know Austrian geography to follow the argument. The only important point is that this node sits between several relevant grid directions.

APG describes Pongau as a newly constructed 380/220/110/30 kV substation in St. Johann im Pongau, commissioned in the first quarter of 2025 as part of the Salzburg line. The substation includes 380 kV GIS switchgear, two 380/220 kV transformers that connect the 380 kV and 220 kV grids, and two 380/110 kV transformers that connect the 380 kV and 110 kV grids. Austrian Power Grid, Pongau Substation

Before looking at the substation structure, we should first place it on the map. If you do not know Austrian geography, “Pongau” is just a name. The important thing is that it sits in a region where several relevant directions meet: toward Salzburg, toward Kaprun and Tauern, toward Weißenbach, and toward the regional Salzburg grid.

Figure 6: Pongau on the Austrian transmission map.

Pongau is not just a transformer in a field. It is a grid node where different voltage levels, transmission directions, pumped-storage connections, and regional distribution meet.

This is not a transformer in a field.

At this one node, different voltage levels meet. A 380 kV system from the Salzburg line is integrated. A connection continues toward Kaprun and Tauern. The 220 kV side connects toward Weißenbach. The 110 kV side connects to Salzburg Netz and the regional distribution grid. The 30 kV level is local or auxiliary in this simplified view, not a major transmission corridor.

This is exactly why the graph model is useful.

If we tried to explain Pongau with the pipe analogy, we would ask the wrong question: where does the electricity come from, and where does it go?

But the better question is: what does this node connect?

It connects voltage levels. It connects transmission paths. It connects regional distribution. It connects pumped storage. It strengthens supply reliability. It becomes part of the high-voltage architecture that allows the Austrian grid to operate as a system rather than as isolated local supply islands.

A substation like Pongau is not a passive object on the way from generator to consumer. It is part of the architecture that makes the whole system usable.

Generators do not “send electricity to houses”

Now that we have the graph, we can attach generators to it.

A generator is any facility that injects electrical power into the grid. It can be a hydropower plant, a gas plant, a wind farm, a solar plant, a biomass plant, or another generation source.

But the important word is injects.

A generator does not choose one household and send energy to it. It pushes power into the synchronized system. That changes the state of the system, and the physical network determines how power flows.

This is why the simple sentence “this power plant supplies this city” is often misleading. It may be true in an administrative, regional, or approximate sense, but electrically the situation is more complex. A power plant is connected to a grid node. Consumers are connected to other grid nodes. The system balances the total injection and withdrawal through the whole network.

Not let’s look at Kaprun.

Kaprun is a small municipality in the Austrian Alps, in the federal state of Salzburg, but in the Austrian power system the name means much more than a tourist village near mountains and reservoirs. The Kaprun area is closely associated with hydropower and pumped-storage infrastructure. It sits near the Alpine water reservoirs and power plants that can turn stored water into electricity when the grid needs power, or consume electricity to pump water back uphill when storing energy makes sense.

That makes Kaprun perfect for this article. It shows why “generator” and “consumer” are not always fixed identities. VERBUND (Austria's leading electricity company and one of the largest producers of hydroelectricity in Europe) describes the Kaprun power plant group as a storage power plant system with turbine capacity and pumping capacity. Pumped storage is useful because it can generate electricity when water flows down through turbines, but it can also consume electricity when pumps move water back up into a higher reservoir.

Figure 8: Pumped storage breaks the simple category. It can inject power like a generator or withdraw power like a large consumer.

This destroys another simple category.

A pumped-storage plant can behave like a generator. And it can behave like a consumer.

When the system needs power, it can inject power. When there is excess generation or when storage is economically and operationally useful, it can withdraw power and store energy as water at a higher elevation.

So even the terms “generator” and “consumer” are not fixed identities forever. They are roles a component can play in the current operating state of the grid.

Consumers are not the end of a pipe

Consumers also become clearer in the graph model.

A house is a consumer. A factory is a consumer. A railway system is a consumer. A city is a huge collection of consumers. A distribution grid can look like one large withdrawal point from the perspective of the transmission grid.

Most individual consumers are not connected directly to the high-voltage transmission grid. They are connected through lower-voltage distribution networks. Those distribution networks are then connected to the transmission grid through substations and transformers.

So when we zoom out to the transmission level, we should not imagine millions of tiny sockets attached directly to the 380 kV grid. We should imagine large withdrawal areas connected through distribution networks.

This is another reason the pipe picture fails.

The grid is not one pipe that becomes smaller and smaller until it reaches your phone charger. It is a layered electrical system. The high-voltage grid forms the backbone. Substations connect it to regional and local networks. Distribution grids bring power closer to consumers. And at each level, voltage, protection, switching, measurement, and operation matter.

The closer we move to individual homes, the more radial the system often becomes. But the transmission grid itself is designed more like a meshed backbone because important regions should not depend on one fragile path.

Redundancy: the graph should not break too easily

In graph theory, a bridge is an edge whose removal disconnects part of the graph.

That idea is useful for thinking about the transmission grid.

A fragile grid would have critical lines where one failure separates a whole region from the rest of the system. A stronger grid gives important regions more than one way to stay connected. Real grids are not perfectly bridge-free everywhere, but redundancy is one of the key design goals of transmission infrastructure.

Figure 9: In a fragile graph, one failed edge can disconnect a region. A stronger transmission grid creates alternative paths.

Austria gives a good example here. APG describes the 380 kV Salzburg line as part of Austria’s extra-high-voltage ring, and says the ring structure allows power to flow to customers from either direction. The Salzburg line closed a gap in the western part of that ring and entered full operation in April 2025.

The same idea appears in southern Austria. APG’s Carinthia power grid area project plans a 380 kV connection between Obersielach and Lienz to close the 380 kV grid in southern Austria. APG explains that the 380 kV ring creates a redundant connection because important substations can be supplied from two sides.

This is the graph model becoming real engineering.

We build redundancy because real systems fail. Lines are taken out for maintenance. Weather damages infrastructure. Equipment trips. Loads change. Generation patterns shift. If the grid were a simple tree, every important edge would become a potential disaster.

A meshed high-voltage grid gives operators more room to keep the system alive.

Again, we should not confuse this with the internet. The grid does not reroute power like packets. But topology matters. Connectivity matters. Alternative paths matter. A ring is not just a geometric shape. It is a way to reduce dependence on a single corridor.

So what is the grid?

At this point, we can rebuild the picture.

The grid is not a pipe. It is not a chain from power plant to socket. It is not a delivery system. It is a synchronized graph built from physical infrastructure.

High-voltage transmission lines form the backbone. Substations are the nodes where lines, transformers, voltage levels, distribution grids, generators, and storage systems connect. Generators inject power into the system. Consumers withdraw power from it. Pumped storage can do both. Redundancy helps the system survive outages and maintenance. Operators and protection systems keep the graph inside safe limits.

And all of this is part of a larger synchronized machine.

The most important idea is not that the grid has many components. The most important idea is that all these components participate in one continuously changing operating state.

Generation changes. Consumption changes. Storage changes mode. Power flows change. Lines become loaded or unloaded. Substations connect and disconnect parts of the graph. Protection systems wait for faults. Operators watch the system.

And through all of that, the grid must stay balanced.

The real goal: balance

The whole grid has one brutal rule: generation and consumption must stay balanced all the time.

But neither side is stable.

Consumers switch devices on and off. Factories start machines. Trains accelerate. Cities move through morning peaks and evening peaks. Wind farms produce more or less depending on the weather. Solar production changes with clouds and time of day. Power plants start, stop, ramp up, or reduce output. Pumped-storage plants can suddenly become large consumers or large generators.

And still, the system has to remain synchronized.

Across Austria, across neighboring countries, across the continental European grid, the machine has to keep its frequency close to 50 Hz. APG’s balancing information describes 50 Hz as the set point value in the Continental Europe synchronous area and explains that generation and consumption must stay in constant equilibrium. APG Balancing

That is the real goal of the grid.

Not simply to “send electricity” from power plants to houses.

The real goal is to continuously adapt generation, consumption, storage, topology, voltage, and power flows so the whole system remains stable.

This is why the pipe analogy is not only incomplete. It hides the most beautiful part of the system.

A pipe does not need synchronization. A pipe does not have frequency. A pipe does not react to every change in the whole continent. A pipe does not have to balance generation and consumption in real time.

The grid does.

And now that we can see the grid as a synchronized graph, the next question becomes much more interesting:

How is this machine operated?

In the next article, we will look at grid operation and see how all the components we discussed, transmission lines, substations, generators, consumers, storage systems, and control systems, are used to keep generation and consumption balanced in real time.

Sources

• Austrian Power Grid: [Power Grid Austria]

• Austrian Power Grid: [Pongau Substation]

• Austrian Power Grid: [Salzburg Line]

• Austrian Power Grid: [Carinthia Power Grid Area]

• APG Market: [Balancing]

• ENTSO-E: [Transmission System Map]

• VERBUND: [Power plants and hydropower information]

Most requirements in critical industries come from the physical world, and the power grid is no exception. Electricity has rules that cannot be negotiated with. Distance, geography, weather, forests, cities, mountains, sea cables, and even animals shape how the grid must be built and operated.

Before we talk about energy companies, markets, software, smart meters, substations, or regulation — we need to understand the physical system underneath. The grid is not just a business system for buying and selling energy. It is a large physical machine that must stay stable every second.

My approach is simple: deconstruct and rebuild. We start with the smallest useful ideas, understand what problem each one solves, and then follow how they connect into the larger system.

So let’s start with electricity itself.

Subscribe now

The Water Analogy Is Too Small

A good analogy can make a complicated subject click. A wrong one can quietly block the whole understanding — for a very long time.

This happened to me with electricity.

The common explanation says that electricity is like water flowing through pipes. Voltage is pressure, current is flow, wires are pipes, and a load is something that uses the flow.

At the beginning, this helps. It gives you a picture for a simple circuit. A battery pushes, current flows, a lamp lights up, a heater becomes warm. Nice.

But later this picture becomes a wall.

It does not explain motors. It does not explain generators. It does not explain transformers. It does not explain why AC exists, why the grid uses high voltage, why three phases, or why frequency matters so much. And the frustrating part is not that those topics are hard — the frustrating part is that the picture in your head has no room for them. You try to apply the analogy and it just... does not fit. So you feel stupid. But you are not stupid. The analogy is just too small.

Electricity is not only something flowing inside wires. A wire with current is part of an electromagnetic system. Charges move in the conductor, but electric and magnetic fields exist around it — and those fields are not a side detail. They are the reason motors, generators, transformers, AC grids, and frequency make any sense at all.

So: keep the water picture for the first few steps. Then throw it away and build a new one.

The grid is not a pipe system. It is an electromagnetic machine.

And I want to give you a set of analogies that actually maps to this. Not one Swiss Army Knife analogy that is universally bad at everything. A whole zoo of weird, specific, memorable animals — each responsible for one idea, each vivid enough that you cannot forget it.

Let’s go.

Voltage, Current, Power, and Resistance (Where Pipes Still Help)

Here is where the pipe analogy earns its last few minutes on stage.

A source creates an electrical push between two points. This push is voltage. When we connect a load and close the circuit, charge can move — and this movement of charge is current.

The load is where energy becomes something useful. Heat in a heater. Light in a lamp. Motion in a motor. Computation in electronics.

Power tells us how much energy is delivered per second:

Power = Voltage × Current P = V × I

This formula matters because power is not only voltage and not only current. It is both together. High voltage with almost no current does not deliver much power. High current with very low voltage also does not necessarily deliver much. Useful electrical power comes from the combination.

Resistance is the opposition a material gives to current. When current flows through resistance, part of the energy becomes heat. In a heater — exactly what we want. In a transmission line — exactly what we do not want.

This already gives us one of the main grid problems: we need to move huge amounts of energy over long distances without turning the wires into giant heaters.

But to understand the solution, we first need the piece that the water analogy keeps hiding.

Current Creates a Magnetic Field (The Piece That Changes Everything)

When current flows through a wire, a magnetic field appears around that wire.

Stop here for a second. This is not a small detail. This is the idea that unlocks everything else in the article.

The field is not made from tiny pieces of magnet escaping through the insulation. Electrons are not little magnetic balls flying outward. Nothing leaves the wire.

The idea is stranger and more useful: moving electric charge creates a magnetic field around the path where it moves. That is all. Current flows, field appears around the wire. Stronger current, stronger field. Current changes direction, field changes direction.

A compass near a current-carrying wire reacts — not because anything jumped out of the wire, but because the wire changed the space around it.

This is the moment where the water analogy breaks completely. Water flowing through a pipe stays inside the pipe. Current in a wire creates an effect outside. Electricity can influence the world without electrons leaving the conductor.

This is the bridge to motors, generators, and transformers. Everything from here is built on top of this one idea.

Motors and Generators Are the Same Story — In Two Directions

A motor does not work because electrons hit a wheel the way water hits a turbine.

A motor works because magnetic fields push and pull on each other.

Imagine a coil of wire. When current flows through it, the coil behaves like an electromagnet. Place it near another magnetic field, and forces appear. Those forces can create torque. Torque can rotate something.

But here is the problem: one fixed magnetic push is not enough for continuous rotation. If the field stays the same, the rotor moves until it reaches a comfortable position — and then it stops. The same force that helped it move now holds it there. Like a magnet stuck to your fridge door.

So the trick is not only to create a magnetic field. The trick is to keep changing it at the right moment.

Change the direction of current — the field changes direction too. Now the force continues to pull and push the rotor forward instead of letting it settle. Control the timing well enough and you get smooth, continuous rotation.

This is the basic idea behind electric motors: controlled current → controlled magnetic fields → motion.

A generator is the same story in the opposite direction.

In a motor, electricity creates motion. In a generator, motion creates electricity. Move a magnet near a coil, or move a coil through a magnetic field — the magnetic field through the coil changes. A changing magnetic field creates an electric field in the conductor, and that electric field pushes charges. The result is voltage.

A generator does not pour electrical liquid into a wire. It creates voltage because motion changes magnetic fields.

And because large generators rotate, the voltage they create naturally rises, falls, crosses zero, reverses direction, and repeats. Which brings us to AC.

AC Is Not Useless Back-and-Forth Movement

DC means direct current. The push keeps the same direction. A battery is the simplest example.

AC means alternating current. The voltage rises, falls, crosses zero, reverses, rises in the opposite direction, and repeats. In Europe, the grid does this 50 times per second — 50 Hz.

At first, AC feels wrong. If current goes one way and then back again, how does it deliver energy? Doesn’t it cancel itself?

The water picture misleads us here again.

Think of a saw. It moves back and forth, but it cuts wood. A bow drill moves back and forth, but it creates enough heat to start a fire. The motion does not need to travel forever in one direction to do work. The change itself is what does the job.

In AC systems, charges mostly move back and forth locally. The electromagnetic field transfers energy through the system, and loads take energy from that field. Electrons do not need to travel from a power plant to your laptop. The field moves through the system, and energy moves with it.

AC is useful precisely because it keeps changing. Changing fields are created naturally by rotating generators. Changing fields drive motors. And changing fields make transformers possible — which turns out to be the key to building any large-scale grid.

High Voltage and Transformers Solve the Distance Problem

Now we can come back to the problem we parked earlier: moving energy far without losing most of it as heat.

The wire between a generator and a city has resistance. Current flowing through resistance turns energy into heat. The formula is:

Line losses = Current² × Resistance P_loss = I² × R

That square is the dangerous part. If current doubles, losses become four times larger. If current becomes ten times larger, losses become one hundred times larger.

Current is expensive. It heats wires, wastes energy, and forces us to build thicker, heavier, and more costly infrastructure.

But power is voltage multiplied by current. For the same amount of power, we can choose different combinations: lower voltage and higher current, or higher voltage and lower current. For long distances, the second option wins easily.

Raise the voltage. Lower the current. Reduce the losses.

This is why high-voltage transmission lines exist. Not because engineers love dangerous numbers. It is because without high voltage, long-distance transmission is just a very expensive heater.

But high voltage that is useful for transmission is not something you want coming out of a wall socket. So the grid needs a machine that can raise voltage before the long journey, and lower it again at the destination.

This machine is the transformer.

A transformer has two coils wrapped around a magnetic core. One coil is connected to one circuit, the other to another circuit. The two coils are not directly connected by a wire. Electrons do not flow from the first coil into the second.

And still, energy moves across.

With the water analogy, this looks like magic. With fields, it becomes completely clear:

AC in the first coil

→ changing magnetic field in the core

→ changing field reaches the second coil

→ voltage is induced in the second coil

→ power is delivered to the load

The field connects the two circuits.

This also explains why a normal transformer needs AC. Connect steady DC to the first coil and there is only one brief moment — when the current starts — where the field changes. After that, the field becomes steady. A steady magnetic field does not continuously induce voltage in the second coil. No change, no transformer action.

The transformer can also change the voltage level because the two coils can have different numbers of turns:

V_secondary / V_primary ≈ N_secondary / N_primary

More turns on the second coil — voltage steps up. Fewer turns — voltage steps down.

Now the skeleton of the grid is visible. Generators produce power. Step-up transformers raise the voltage. High-voltage lines carry it over long distances. Substations connect, protect, switch, and transform. Step-down transformers bring voltage back down. Distribution networks bring electricity closer to consumers. Loads turn energy into heat, light, motion, or computation.

But there is still one physical improvement missing.

Why the Grid Uses Three Phases

We have AC, transformers, and high-voltage transmission — but one important piece is still not quite right for large-scale use.

Remember the motor problem: we do not only want a magnetic field. We want a magnetic field that keeps the rotor moving smoothly. With a single AC wave, the push changes, weakens, crosses zero, reverses, comes back. It works, but the motor has to fight through those weak and zero points in every cycle.

The better idea: use three AC waves instead of one. Not three random waves — three coordinated waves, shifted in time relative to each other. When one phase is near its maximum, the others are at different points in the cycle. When one weakens, another is already growing stronger. Together, they fill in each other’s gaps and create a much smoother delivery of power.

For motors, this is especially valuable because three coordinated phases can create a naturally rotating magnetic field. Not a push that has to fight through weak spots — a field that smoothly turns around the motor and pulls the rotor with it. This is almost exactly what a motor wants.

Now, the elegant part: we do not need six wires to run three separate circuits.

Each phase needs one conductor — so we have three phase conductors: L1, L2, and L3. In a balanced system, the currents in these three phases are shifted so that their sum is zero at every moment. Because of that, the return current cancels out, and a separate return wire is not needed for high-voltage transmission.

This is why you often see three main conductors on transmission towers — not three complete circuits with six wires, but one coordinated three-phase system using three.

Three phases are not an arbitrary complication. They solve a physical problem: smoother power delivery, better conditions for motors, and more efficient large-scale transmission with fewer conductors.

Frequency Is the Pulse of the Grid

Now we have generators, AC, transformers, high voltage, and three phases. The physical machine is assembled.

But the grid is not a tank where you can pour electricity in and take it out freely whenever you want. There is some stored energy in rotating machines, magnetic fields, batteries, and other devices — but the AC grid itself must be balanced continuously. Continuously.

Every second, generation and consumption need to stay close to each other.

If consumers take more power than generators provide, the system starts to slow down. If generation exceeds consumption, the system starts to speed up. And this shows up in one very visible number: frequency.

In Europe, the grid runs at 50 Hz. That means the AC waveform completes 50 full cycles every second. When generation and consumption are balanced, frequency stays close to 50 Hz. If consumption exceeds generation, frequency falls. If generation exceeds consumption, frequency rises.

Frequency is not just a number in a technical standard. It is one of the signs that shows the balance of the whole synchronized machine. It is the pulse of the grid.

And this is where the power grid becomes very different from most software systems. In software, we can queue work, buffer messages, retry requests, scale components, and hide temporary imbalance behind storage. The grid has controls and buffers too — but the physical system still has to obey the immediate relationship between generation, load, voltage, current, phase, and frequency. You cannot queue it. You cannot retry it.

The grid is an electromagnetic machine spread over a continent, and that machine must stay inside its limits. If the pulse is stable, the system is alive and balanced. If the pulse drifts too far, something is wrong.

That is the grid’s arrhythmia.

And you have to heal it!

What We Have Now And What Is Next

We started with the water analogy because it helps in the beginning. It explains voltage, current, resistance, power, and simple circuits.

But then it breaks.

It cannot explain how electricity creates motion outside a wire. For that, we need fields.

Current creates magnetic fields. Magnetic fields push and pull. Controlled magnetic fields create motion — motors. Reverse the story: motion changes magnetic fields, changing fields create voltage — generators.

Then AC stops being strange. It is not useless back-and-forth. It is repeated change, and changing fields are exactly what generators, motors, and transformers need.

Then distance creates the next problem. To move power far away, we reduce current and increase voltage. Because high voltage is not useful everywhere, we use transformers to step it up and back down.

Then three phases make the system smoother — better for motors, smoother power delivery, efficient transmission without six wires for three circuits.

And finally, frequency shows us that the grid is not a passive network of wires. It is a synchronized machine that must stay balanced every second.

The water analogy was not completely wrong. It was just too small. It helped me enter the first room, and then it locked all the next doors.

The field-based model opens them.

It does not make the grid simple, because the grid is not simple. But it makes the complexity connected. Motors, generators, transformers, AC, high voltage, three phases, and frequency stop being random technical vocabulary and become parts of the same engineering story.

In the next part, we can move from physics to structure: power plants, substations, transmission lines, distribution networks, and the full path from generation to consumption.

If you want to follow this rebuild, subscribe and the next part will find you when it is ready.

I have spent a lot of time learning technical systems by rebuilding them from first principles.

With TLS, I tried to understand the protocol not by starting from the final polished design, but by beginning with a simple and broken version. Then I added one missing idea after another: encryption, integrity, key exchange, certificates, and identity. Step by step, the real structure started to make more sense.

Now I want to apply a similar mindset to another system, but on a much larger scale: the European power grid.

The grid is not a small topic. It is not only wires, power plants, and substations. It is also physics, real-time operation, markets, regulation, international coordination, forecasting, automation, cybersecurity, and software. It is one of the most critical systems around us, but for many software developers it often remains hidden behind abstract words like “energy sector,” “smart grid,” “TSO,” “DSO,” “balancing,” “SCADA,” or “grid modernization.”

While trying to learn this field, I noticed a recurring problem: many explanations are either too shallow, too regulatory, or too electrical-engineering-heavy. Some give a high-level business overview, but do not explain the physical system underneath. Others focus on laws, institutions, and market rules, but are hard to connect to the actual grid. Electrical engineering resources often go deep into theory, which is valuable, but not always the easiest entry point for software developers who first need a connected map of the whole system.

That is the gap I want to work through.

I want to build a central learning hub where I slowly connect these pieces together.

In this post I will explain what I want to build, why I am building it, and how I want to approach the European power grid from the perspective of a software developer who wants to understand the industry from the ground up.

Subscribe now

Why I Am Starting This

My goal is simple: as a software developer, I want to better understand this critical industry and the components it is built from.

This is not a random topic for me. I already work in this industry, and the closer I get to energy systems, grid-related software, and operational technology, the more I see that software is only one layer of a much larger system. To build better software, make better technical decisions, and understand the real problems behind the requirements, I need a clearer picture of the whole grid.

I do not want to look at energy software as isolated applications detached from the physical system underneath. I want to understand what the software is connected to, what constraints the grid has, why those constraints exist, and how the different layers of the industry fit together.

When we build software for ordinary web applications, we can often reason mostly in terms of users, APIs, databases, queues, services, and infrastructure. The physical world still matters, but it is often abstracted away.

The power grid is different.

Software in this industry does not exist in a vacuum. It is connected to physical equipment, operational constraints, safety requirements, regulations, market structures, and cross-border coordination. A wrong assumption is not just a wrong assumption inside an application. It can touch real infrastructure, real operators, real assets, and real consequences.

That makes the field difficult, but also very interesting.

I want to understand the grid not as a collection of disconnected terms, but as one layered system. I want to see how electricity behaves, how the physical infrastructure implements that behavior, how operators keep the system stable, how organizations divide responsibility, how Europe coordinates the whole system, and where software enters the picture.

The Problem With Existing Explanations

When I started looking for good explanations of the European power grid, I often found useful pieces, but not a full map.

Some resources explain the energy sector from a very high level. They describe renewables, transmission, distribution, markets, and policy, but they often stay too far away from the physical system. After reading them, you may know the names of the actors, but still not understand what is actually happening in the grid.

Other resources are very regulatory. They explain institutions, market rules, network codes, responsibilities, tariffs, and legal structures. That is important, but if you do not already understand the physical and operational system, it is hard to know where those rules attach to reality.

Then there are electrical engineering resources, which are often much deeper technically. They explain circuits, machines, power systems, load flow, protection, and control theory. These resources are valuable, but they are not always the right first step for a software developer who wants to build a practical mental model before going deeper into equations and specialist theory.

So the problem is not that the information does not exist.

The problem is that it is fragmented across different worlds.

There is the physics world. There is the electrical engineering world. There is the operator world. There is the regulatory world. There is the market world. There is the software world. Each of them has its own language, assumptions, and priorities.

What I want to build is a bridge between those worlds.

Who This Is For

I am writing this from the perspective of a software developer, but I do not think the topic is useful only for software developers.

Many people enter the energy industry from different angles. Some come from cloud engineering, testing, cybersecurity, data engineering, product management, business analysis, operations, regulation, consulting, or management. Many of them face the same problem: they see parts of the system, but not the whole picture.

Even if my own final focus will be software, I believe that a clearer map of the whole grid can be useful for anyone who wants to understand where their work fits.

If you work on an energy platform, it helps to know what physical process your data represents.

If you work on testing, it helps to know which failures matter operationally.

If you work on cybersecurity, it helps to understand what assets, protocols, and control paths are critical.

If you work in cloud or data, it helps to know why not every problem in this industry can be treated like a normal SaaS problem.

If you work in product or management, it helps to understand the difference between a market requirement, an operational requirement, and a physical constraint.

That is the kind of map I want to build.

Not a complete electrical engineering education.

Not a regulatory encyclopedia.

Not a shallow overview of energy trends.

A practical, layered explanation of the European power grid for people who want to understand the system behind the software, decisions, and infrastructure.

What This Hub Will Become

This page will become the central hub for the series.

I will update it as new articles are published, and over time it should become a structured entry point into the whole topic. The goal is not to explain everything at once. The goal is to build the map piece by piece.

First, electricity itself.

Then the physical grid.

Then operations.

Then management.

Then governance and markets.

Then international coordination.

Then software.

I expect this to take time, and that is fine. Complex systems are not understood in one article. They become understandable when we build the right layers in the right order.

That is what I want to do here.

I want to take the European power grid, one of the most critical systems around us, and make it more understandable from a developer’s point of view.

Not by pretending it is simple.

But by building the mental model step by step, from physics to software, until the grid starts to feel less like a black box and more like a system that can be understood.

All parts of the series:

If you are interested in how the European power grid works, from electricity and infrastructure to operations, governance, and software, subscribe to follow the series as it grows.

Previous Article:

A secure connection to the wrong server is still a broken connection.

That sentence looks strange at first. If traffic is encrypted, if nobody can read it, if nobody can modify it, then what is still missing?

The missing piece is identity.

In the previous parts of this series, we slowly moved from a plain TCP connection to something that started to look like a real secure channel. First, we added encryption. Then we added integrity. Then we stopped using fixed shared keys and introduced a handshake with X25519 and HKDF, so the client and server could derive fresh session keys for every connection.

That was a big step forward, but it was still not enough.

The client could now create a strong encrypted channel, but it still had no reliable way to know who it had created that channel with. It could be the real server. It could also be an attacker sitting in the middle, performing a separate key exchange with the client and another one with the real server.

In this post we will discuss why key exchange is not the same as authentication, why certificates exist, how certificate chains work, and how I added a simplified certificate-based authentication layer to my small TLS-like protocol.

By the end, the protocol will finally move from:

“I have an encrypted channel with whoever answered.”

to:

“I have an encrypted channel with the server that proved its identity.”

That is the moment where this toy protocol starts to feel much closer to real TLS.

Subscribe now

The Problem We Still Had After Key Exchange

At the end of Part 3, the protocol already had a real handshake.

The client and server exchanged ephemeral X25519 public keys. They used the resulting shared secret as input to HKDF. From that, they derived separate keys for client-to-server and server-to-client traffic. Then they used those keys to protect application data with AES-GCM.

That is already much better than hardcoding a shared key into both programs.

A hardcoded key has many problems. If someone gets it once, every connection using that key is compromised. If you want to rotate it, you need to update both sides. If two clients use the same key, one compromised client can affect other connections. It is simple for a demo, but it is not a good model for a real secure protocol.

X25519 and HKDF solved a different problem. They allowed the client and server to create fresh session keys for every connection without sending the actual secret over the network.

But there was still a hole.

The client received a public key from “the server,” but it had no way to know whether that public key really belonged to the server. The protocol could protect traffic after the handshake, but the handshake itself was not authenticated.

That means a man-in-the-middle could sit between the client and server and do this:

Client  <---- key exchange ---->  Attacker  <---- key exchange ---->  Server

From the client’s point of view, everything looks fine. It completed a key exchange and derived keys.

From the server’s point of view, everything also looks fine. It completed a key exchange and derived keys.

But the attacker is in the middle of both secure channels.

This is the uncomfortable lesson:

A secure key exchange with an unauthenticated peer can still give you a secure channel to the attacker.

That is why Part 4 exists.

Encryption Is Not the Same as Trust

It is easy to mix these ideas together because HTTPS makes them feel like one thing.

When we open a website over HTTPS, we usually think:

The connection is encrypted, so it is secure.

But real TLS gives us several properties at the same time. Encryption is only one of them.

A useful way to separate the ideas is this:

Encryption answers:
Can outsiders read the data?

Integrity answers:
Can outsiders modify the data without being detected?

Key exchange answers:
Can both sides create fresh session keys?

Authentication answers:
Do I know who is on the other side?

Before Part 4, our protocol had the first three. It did not have the fourth.

That distinction matters because an attacker does not always need to break encryption. Sometimes the attacker only needs to become the endpoint that you encrypt to.

If the client encrypts data to the attacker’s key, the encryption still works. The math is not broken. AES-GCM still protects the records. HKDF still derives keys. X25519 still creates a shared secret.

The problem is not the cryptography. The problem is that the client trusted the wrong public key.

So the next question becomes simple:

How does the client know that the server’s handshake key belongs to the real server?

This is where certificates enter the story.

What Certificates Actually Add

A certificate is not magic. It is also not just a random file that makes browsers happy.

At a practical level, a certificate connects an identity to a public key.

For example, a server certificate says something like:

This public key belongs to this server identity.

But the client should not just believe that statement because the server said so. Anyone can generate a key pair and create a file that claims to be example.com.

So certificates are signed.

That means another key, usually belonging to a Certificate Authority, signs the certificate data. The client can then verify the signature using the Certificate Authority’s public key.

In the real Web PKI, this usually forms a chain:

Root CA
  signs
Intermediate CA
  signs
Server Certificate

The root certificate is already trusted by the client’s system or browser. The server sends its certificate chain during the handshake. The client verifies each signature in the chain until it reaches a trusted root.

In my simplified implementation for Part 4, I use the same conceptual model:

Root CA
  ↓
Intermediate CA
  ↓
Server Certificate

The point is not to recreate all of Web PKI. The point is to make the trust chain visible.

The client does not trust the server certificate because the server sent it. The client trusts it because it can verify that the certificate was signed through a chain that ends at a trusted root.

Now the client has something it did not have before:

A public identity key that is connected to the server certificate and can be verified through a certificate chain.

But there is still one more important step.

Why the Server Must Sign the Handshake

A certificate chain proves that a certificate is valid.

It does not automatically prove that the server currently speaking in this connection owns the private key for that certificate.

That difference is important.

If the server only sends a certificate chain, an attacker could potentially copy that public certificate chain and send it to the client. Public certificates are public. They are not secrets.

So the server must prove ownership of the corresponding private key.

In real TLS, this is done with a handshake signature. In TLS 1.3, the server signs data derived from the handshake transcript. This binds the server’s authenticated identity to the exact handshake that is happening now.

In my simplified Part 4 implementation, I use the same core idea, but in a smaller form.

The server has two different types of keys:

Long-term identity key
= connected to the server certificate

Ephemeral X25519 key
= used only for this connection’s key exchange

The server sends its ephemeral X25519 public key, its certificate chain, and a signature over the handshake data.

The client verifies three things:

1. Is the certificate chain valid?

2. Does the server certificate contain the expected identity key?

3. Did the server sign this handshake using the private key that matches the certificate?

This is the key conceptual step.

The ephemeral X25519 key gives us fresh session keys. The certificate gives us identity. The signature connects both worlds together.

Without that signature, the certificate and the key exchange are two separate facts.

With the signature, the server says:

I own the private key for this certificate, and I am binding that identity to this ephemeral key exchange.

That is what stops the man-in-the-middle from silently replacing the handshake key.

The Architecture of Part 4

After Part 4, the handshake looks roughly like this:

Client
  |
  |  ClientHello
  |  ephemeral X25519 public key
  |
  v
Server
  |
  |  ServerHello
  |  ephemeral X25519 public key
  |  certificate chain
  |  handshake signature
  |
  v
Client

Then both sides derive the shared secret using X25519:

client private key + server public key
server private key + client public key

Both sides arrive at the same shared secret without sending that secret over the network.

Then HKDF turns that shared secret into actual session keys.

Then AES-GCM protects the application records.

The important change is that the client no longer accepts the server’s ephemeral public key blindly. It checks whether the authenticated server signed the handshake.

The complete shape now looks like this:

Certificate chain
  proves server identity

Handshake signature
  binds server identity to the ephemeral key exchange

X25519
  creates a fresh shared secret

HKDF
  derives directional session keys

AES-GCM
  protects application records

This is still a simplified protocol, but the main pieces now line up with the real TLS story much better.

What We Built in Code

For this part, I added a small certificate infrastructure to the project.

The implementation generates a simple hierarchy:

Root CA
Intermediate CA
Server Certificate

The server uses the server certificate and private key as its long-term identity. During the handshake, it still creates a fresh ephemeral X25519 key pair for the current connection.

That distinction matters.

The long-term certificate key is not used to encrypt application data. It is used to prove identity.

The ephemeral X25519 key is not used as identity. It is used to create a fresh shared secret for this connection.

This separation is one of the most important design ideas in modern TLS. Long-term keys authenticate. Ephemeral keys protect the session.

The simplified Part 4 flow is:

1. Client creates an ephemeral X25519 key pair.

2. Client sends its public key.

3. Server creates an ephemeral X25519 key pair.

4. Server sends:
   - its ephemeral public key
   - certificate chain
   - signature over handshake data

5. Client verifies the certificate chain.

6. Client verifies the handshake signature.

7. Both sides derive the shared secret with X25519.

8. Both sides derive session keys with HKDF.

9. Application data is protected with AES-GCM.

The full code is in the GitHub repository:

https://github.com/DmytroHuzz/rebuilding_tls

The full technical walkthrough is here:

https://dmytrohuzz.github.io/rebuilding_tls/part_4/walkthrough/walkthrough.html

I intentionally keep the article focused on the protocol idea rather than pasting the whole implementation here. Long code blocks inside an article often create an illusion of depth, but they usually make the article harder to read.

The repository is the right place for the full implementation. The article is the right place for the explanation.

What This Still Is Not

This is not real TLS.

That sentence is important.

This project is an educational reconstruction of some core TLS ideas. It is not a library. It is not a production protocol. It is not something that should protect real traffic.

Real TLS has many more details and much stronger guarantees around the handshake. For example, real TLS 1.3 binds the handshake with a transcript hash. It supports negotiation, extensions, certificate validation rules, revocation mechanisms, session resumption, alerts, many edge cases, and years of hardening against attacks that are easy to miss when building a small protocol.

My version is intentionally smaller.

It is useful because it makes the main ideas visible:

Why encryption alone is not enough.

Why integrity must be added.

Why fixed keys are weak.

Why key exchange gives fresh session keys.

Why key exchange is still not authentication.

Why certificates exist.

Why the server must sign the handshake.

Why TLS has the shape it has.

That is the real goal of this series.

Not to replace TLS, but to make TLS less mysterious.

Try It Yourself

The project is public and runnable.

The main repository is here:

https://github.com/DmytroHuzz/rebuilding_tls

The Part 4 walkthrough is here:

https://dmytrohuzz.github.io/rebuilding_tls/part_4/walkthrough/walkthrough.html

The complete series landing page is here:

If you want to understand the path from the beginning, I recommend reading the series in order. Each part fixes one problem and reveals the next one.

Part 1 starts with encryption. Part 2 adds integrity. Part 3 adds key exchange and session keys. Part 4 adds authentication through certificates and a handshake signature.

That sequence matters because it shows why TLS is not just “encryption.” It is a stack of answers to different problems.

Summary

Before Part 4, the protocol could create an encrypted channel, but it could not prove who was on the other side.

That is a serious problem.

Encryption protects data from being read. Integrity protects data from being modified. Key exchange creates fresh session keys. But authentication tells the client whether it is talking to the real server or to an attacker in the middle.

In Part 4, I added that missing authentication layer.

The client now verifies a certificate chain and checks a handshake signature. The server uses a long-term identity key to authenticate itself, while still using an ephemeral X25519 key for the actual key exchange. HKDF derives session keys, and AES-GCM protects application data.

The result is still not real TLS, but it now has the core shape:

authenticated handshake
fresh session keys
protected records

And that is the point of this whole series.

TLS is hard to understand when you only look at the final protocol. There are too many details, too many names, too many moving parts.

But when you rebuild it step by step, each piece starts to make sense.

You first feel the problem.

Then you add the missing mechanism.

Then you see why the real protocol looks the way it does.

Hi, I’m Dmytro.

This publication is about understanding complex engineered systems by taking them apart, tracing how their layers fit together, and sometimes rebuilding important pieces from scratch.

If you are new here, this page is the easiest place to begin.

You do not need to read everything in chronological order. Pick the system that interests you and go from there.

Subscribe now

Rebuilding technology from first principles

This is where the publication started.

I wanted to understand technologies I had used for years but never truly understood underneath their abstractions.

So I started rebuilding them.

Rebuilding a web server

A web server looks simple from the outside: receive a request, return a response.

Underneath that simple interaction are sockets, TCP, HTTP parsing, concurrency, operating-system interfaces, and a surprising number of small engineering decisions.

This project was my attempt to follow the path all the way down and build a small server myself.

Start here:
https://dev.to/dmytro_huz/building-your-own-web-server-part-1-theory-and-foundations-3kgo

Rebuilding cryptography

TLS eventually forced me into another layer.

Before I could understand secure communication properly, I needed to understand the cryptographic primitives underneath it.

So I explored things like encryption, hashing, key exchange, authentication, and the mathematics that makes them possible, then implemented simplified versions myself.

Start here:

Rebuilding TLS from scratch

This became one of my longest projects.

I spent roughly a year and a half learning how TLS works, rebuilding its important parts, and eventually connecting my own toy TLS implementation to my own web server.

The goal was never to create production cryptography.

It was to understand what really happens between typing https:// and establishing a secure connection.

If you want one project that best represents the original idea behind Rebuilt, start here.

Start here:

Understanding time in computers

Time sounds trivial until distributed computers need to agree on it.

Then you encounter oscillators, clocks, drift, operating systems, synchronization, networks, NTP, precision, and the uncomfortable realization that even “what time is it?” is a systems problem.

Start here:

Understanding larger systems

At some point, I became interested not only in rebuilding individual technologies but in understanding systems much larger than a single protocol or application.

The method remained the same:

start with the fundamentals, identify the layers, understand how they interact, and gradually reconstruct the whole mental model.

The electrical grid for software engineers

The power grid became my first attempt to do that at infrastructure scale.

I work as a software engineer in the energy sector, but I initially understood surprisingly little about the system around the software.

So I started from the beginning.

Why does grid frequency matter?

How does electricity actually move?

What do transmission and distribution operators do?

How is the grid controlled?

Where do energy markets enter the picture?

And where, across all those layers, does software actually live?

The result became a series explaining the European electrical grid from a software engineer’s perspective.

A good reading order

1. A Developer’s Map of the European Electrical Grid
   The overall mental model and the major layers of the system.

2. The Grid Is Not a Pipeline
   The physical foundations and why electricity behaves differently from most resources we move around.

3. How the Grid Is Operated
   Frequency, balancing, reserves, control, and the mechanisms keeping the system stable.

4. How the Electricity Market Works
   Producers, consumers, BRPs, exchanges, system operators, schedules, settlement, and why the commercial system exists.

5. Where Software Lives in the Power Grid
   How software appears across field devices, substations, control centers, markets, analytics, enterprise systems, and everything between them.

Start here:

What I want to build next

The projects above started at opposite ends.

One began with individual technologies such as HTTP and TLS.

The other began with an enormous physical infrastructure system.

Now I am increasingly interested in the space where those two directions meet.

Systems that combine several layers:

• software

• networks

• protocols

• security

• devices

• embedded or edge computing

• physical processes

• operations

• reliability

I want to move beyond explaining isolated components and start building more complete systems myself.

That may mean rebuilding a VPN and eventually putting it into a physical device.

It may mean experimenting with industrial communication protocols.

It may mean connecting software to sensors, field devices, simulators, or real infrastructure.

And it will probably mean discovering several layers I currently know almost nothing about.

That is part of the appeal.

What Rebuilt is about

The subjects here may look quite different.

TLS and electrical grids do not have much in common on the surface.

But the questions behind them are usually the same.

How does the system really work?

Why is it structured this way?

Where are the boundaries?

How do the layers communicate?

What assumptions hold everything together?

What happens when something fails?

And what would I learn if I tried to build part of it myself?

That is the thread connecting everything here.

If you enjoy going underneath abstractions and understanding how complex systems actually fit together, you are in the right place.

Welcome to Rebuilt.

Previous Article:

Overview: Where we are and What Is Still Missing

In the previous part of this series, we made our fake secure channel much less fake.

We started with the broken encrypted transport from Part 1, added integrity with HMAC, added sequence numbers to make the record layer less naive, and then moved to AEAD — the approach modern systems usually use to protect records.

At that point, our protocol could already do something meaningful:

• encrypt application data

• detect tampering

• reject modified records

• keep some minimal record-layer state

That was a real step forward.

But it still relied on one very unrealistic assumption:

both sides already shared the secret keys

And that is exactly what we need to remove now.

Because a real secure protocol cannot stop at protecting data after the keys already exist. It also has to answer one of the harder questions first:

if client and server do not already share a secret, how can they create one over an insecure network in the first place?

That is the goal of this part.

We are going to build the next missing layer of the protocol: the handshake.

The architecture of this step is simple:

Client                           Server
------                           ------
Handshake messages  <--------->  Handshake messages
       |                               |
       v                               v
  shared secret                  shared secret
       |                               |
       +---------> HKDF <--------------+
                    |
                    v
              session keys
                    |
                    v
         protected application data

The idea is to let the connection create fresh key material dynamically instead of starting with a hardcoded application key.

We will implement that in three steps.

First, we will build a handshake with classic Diffie-Hellman, where the shared prime and base are still explicit and visible in the protocol. Then we will replace that version with X25519 to show how modern protocols simplify the same idea. After that, we will use HKDF to derive proper session keys from the raw shared secret.

That will take us one big step closer to the shape of real TLS.

But still not all the way.

Because even if both sides manage to derive the same fresh session keys, one critical problem will remain: they still do not know who is on the other side.

And that is where this part is heading.

A Very Short Note on Public Key Exchange

The basic idea of public key exchange is simple.

Two sides communicate over an insecure network. They exchange some public information. And from that exchange, both sides derive the same shared secret — without ever sending that secret directly over the wire.

That is the key point.

The network can be fully visible.

An observer can see all handshake messages.

But the observer still should not be able to derive the same secret.

That is exactly the kind of mechanism we need now.

Until this point in the series, our protocol always started with a secret that already existed. Public key exchange changes that. It gives the connection a way to create fresh shared key material dynamically.

In this article, I do not want to go deep into the mathematics behind it. I only want to use the core idea as the next building block of the protocol.

If you want the deeper intuition behind why this works, I already wrote about it here:

For now, the main idea we need is this:

• each side contributes its own private value

• both sides exchange some public values

• both sides derive the same shared secret

• that secret can then become the basis for session keys

So let’s build that first in the most explicit way, with classic Diffie-Hellman where the shared public parameters are still visible in the handshake.

Implementation Part 1 — Our First Handshake with Classic Diffie-Hellman

(The whole code can be find here: https://github.com/DmytroHuzz/rebuilding_tls/tree/main/part_3/v1_classic_dh_handshake )

Now let’s build the first real handshake in the series.

I want to start with classic Diffie-Hellman, not because this is the final form we want to keep, but because it makes the mechanics of key exchange much more visible.

In this version, both sides work with the same public parameters:

• a prime p

• a generator g

These values are not secret. In our implementation, the client sends them in the handshake, which makes the whole mechanism more explicit on the wire. That is exactly what I want at this stage. Before we hide the details behind a cleaner modern primitive, I want to make the structure fully visible.

The actual secret material comes from somewhere else:

• the client chooses a private exponent a

• the server chooses a private exponent b

From those private values, both sides compute public values:

• the client computes A = g^a mod p

• the server computes B = g^b mod p

Then they exchange A and B.

And this is the key step:

• the client computes s = B^a mod p

• the server computes s = A^b mod p

Both sides end up with the same shared secret, without ever sending that secret directly over the network.

In diagram form, the handshake looks like this:

Client                                        Server
------                                        ------
choose private a
compute A = g^a mod p

ClientHello(p, g, A)      --------->

                                              choose private b
                                              compute B = g^b mod p

                          <---------          ServerHello(B)

compute s = B^a mod p                           compute s = A^b mod p

That is our first real handshake.

Until now, the protocol always started with a secret key that already existed.

Now the connection itself creates the secret.

That is a major shift.

The raw Diffie-Hellman math

At the lowest level, the core operations are very small. That is one of the nice things about starting with classic Diffie-Hellman: the whole idea is still visible in a few functions.

# RFC 3526 Group 14: 2048-bit MODP prime
DH_PRIME = int(
    "FFFFFFFFFFFFFFFFC90FDAA22168C234C4C6628B80DC1CD1"
    "29024E088A67CC74020BBEA63B139B22514A08798E3404DD"
    "EF9519B3CD3A431B302B0A6DF25F14374FE1356D6D51C245"
    "E485B576625E7EC6F44C42E9A637ED6B0BFF5CB6F406B7ED"
    "EE386BFB5A899FA5AE9F24117C4B1FE649286651ECE45B3D"
    "C2007CB8A163BF0598DA48361C55D39A69163FA8FD24CF5F"
    "83655D23DCA3AD961C62F356208552BB9ED529077096966D"
    "670C354E4ABC9804F1746C08CA18217C32905E462E36CE3B"
    "E39E772C180E86039B2783A2EC07A28FB5C55DF06F4C52C9"
    "DE2BCBF6955817183995497CEA956AE515D2261898FA0510"
    "15728E5A8AACAA68FFFFFFFFFFFFFFFF",
    16,
)

DH_GENERATOR = 2

def generate_private_exponent() -> int:
    return int.from_bytes(os.urandom(32), "big")

def compute_public_value(private: int, g: int, p: int) -> int:
    return pow(g, private, p)

def compute_shared_secret(peer_public: int, private: int, p: int) -> int:
    return pow(peer_public, private, p)

This is the whole core idea in code:

• private exponent stays local

• public value goes on the wire

• shared secret is derived independently on both sides

That is the heart of Diffie-Hellman.

Client side

def client_handshake(sock) -> bytes:
    """Perform the client side of the classic DH handshake.

    The client picks the public parameters (p, g) and sends them to the
    server along with its own public DH value.  The server uses those
    parameters to compute its own public value and sends it back.

    Returns the shared secret as bytes.
    """
    # The client chooses p and g.  These are PUBLIC — not secret.
    # Anyone on the wire can see them, and that is perfectly fine.
    # The security of DH depends on the hardness of the discrete
    # logarithm problem, not on hiding p and g.
    p = DH_PRIME
    g = DH_GENERATOR

    print(f"  Public parameters (chosen by client, sent to server):")
    print(f"    p = {str(p)[:40]}... ({p.bit_length()} bits)")
    print(f"    g = {g}")

    # Step 1: Generate client's private exponent and public value.
    # The private exponent is the ONE thing that stays secret.
    client_private = generate_private_exponent()
    client_public = compute_public_value(client_private, g, p)
    client_public_bytes = int_to_bytes(client_public)

    # Step 2: Send ClientHello with p, g, and our public value.
    # All three are public.  The private exponent is NOT included.
    p_bytes = int_to_bytes(p)
    g_bytes = int_to_bytes(g)

    client_hello = encode_message(
        [
            (TAG_DH_P, p_bytes),
            (TAG_DH_G, g_bytes),
            (TAG_DH_PUBLIC, client_public_bytes),
        ]
    )
    # Step 3: send p, g, and the client’s public value inside ClientHello
    send_record(sock, client_hello)

    # Step 4: Receive ServerHello with the server's public value.
    server_hello_raw = recv_record(sock)
    fields = decode_message(server_hello_raw)
    server_public_bytes = None
    for tag, value in fields:
        if tag == TAG_DH_PUBLIC:
            server_public_bytes = value
    if server_public_bytes is None:
        raise ValueError("ServerHello missing DH public value")

    server_public = bytes_to_int(server_public_bytes)
    print(f"  <- Received ServerHello")
    print(f"  Server public value B:   {hex_preview(server_public_bytes)}")

    # Step 5: Compute the shared secret.
    # shared = B^a mod p = (g^b)^a mod p = g^(ab) mod p
    shared_int = compute_shared_secret(server_public, client_private, p)
    shared_bytes = int_to_bytes(shared_int)

    return shared_bytes

On the client side, the flow is:

1. choose a private exponent

2. compute the public value

3. send p, g, and the client’s public value inside ClientHello

4. receive the server’s public value

5. derive the shared secret

That is the first point in the series where the client does not begin with the application key. It participates in creating it.

Server side

def server_handshake(sock) -> bytes:
    """Perform the server side of the classic DH handshake.

    The server receives p, g, and client_public from the ClientHello,
    uses those parameters to generate its own keypair, and sends its
    public value back.

    Returns the shared secret as bytes.
    """
    # Step 1: Receive ClientHello — parse p, g, and client's public value.
    # The server does NOT assume any particular p or g.  It uses whatever
    # the client proposes.  (In a production system, the server would
    # validate that p is a safe prime and g is a proper generator.
    # We skip that here for clarity.)
    client_hello_raw = recv_record(sock)
    fields = decode_message(client_hello_raw)

    p_bytes = None
    g_bytes = None
    client_public_bytes = None
    for tag, value in fields:
        if tag == TAG_DH_P:
            p_bytes = value
        elif tag == TAG_DH_G:
            g_bytes = value
        elif tag == TAG_DH_PUBLIC:
            client_public_bytes = value

    if p_bytes is None:
        raise ValueError("ClientHello missing DH prime (p)")
    if g_bytes is None:
        raise ValueError("ClientHello missing DH generator (g)")
    if client_public_bytes is None:
        raise ValueError("ClientHello missing DH public value (A)")

    # Deserialize the parameters from bytes.
    p = bytes_to_int(p_bytes)
    g = bytes_to_int(g_bytes)
    client_public = bytes_to_int(client_public_bytes)

    # Step 2: Generate server's private exponent and public value
    # using the p and g received from the client.
    server_private = generate_private_exponent()
    
    # Step 3: Compute server's public value
    server_public = compute_public_value(server_private, g, p)
    server_public_bytes = int_to_bytes(server_public)

    # Step 4: Send ServerHello with our public value.
    # Only B is sent — p and g are already known from the ClientHello.
    server_hello = encode_message(
        [
            (TAG_DH_PUBLIC, server_public_bytes),
        ]
    )
    send_record(sock, server_hello)

    # Step 5: Compute the shared secret.
    # shared = A^b mod p = (g^a)^b mod p = g^(ab) mod p
    shared_int = compute_shared_secret(client_public, server_private, p)
    shared_bytes = int_to_bytes(shared_int)

    return shared_bytes

The server does the mirror image:

1. receive p, g, and the client’s public value

2. choose its own private exponent

3. compute its own public value

4. send that value back in ServerHello

5. derive the same shared secret from the client’s public value

So at the end of the handshake, both sides have the same secret — but that secret was never transmitted directly.

That is the big win.

After this step, the connection can create fresh shared key material dynamically.

That is a much more realistic foundation.

But it is also still awkward.

Not conceptually awkward — educationally this version is very useful — but operationally awkward. We now have explicit p and g in the handshake, which is nice for understanding the mechanism, but clunky for a modern protocol design.

That is exactly why the next step will replace this version with X25519.

Implementation Part 2 — Simplifying the Handshake with X25519

(The whole code can be find here: https://github.com/DmytroHuzz/rebuilding_tls/tree/main/part_3/v2_x25519_handshake )

The classic Diffie-Hellman version was useful because it made the mechanics of the handshake fully visible.

But it also makes something else visible:

it is a bit clunky.

Not conceptually clunky — educationally it is great — but operationally clunky. There are more moving parts in the handshake, more explicit protocol fields, and more visible math than modern protocols usually want to expose directly.

So now we keep the same core idea and simplify the workflow.

That is where X25519 comes in.

The conceptual goal stays exactly the same:

• both sides generate ephemeral private/public key pairs

• both sides exchange public keys

• both sides derive the same shared secret

• that secret will later become the basis for session keys

What changes is the shape of the handshake.

We no longer need to carry an explicit prime and generator through the protocol. We no longer manually perform modular exponentiation with visible p and g. X25519 gives us the same public-key exchange idea in a much cleaner modern form.

That is why I wanted this section right after the classic DH version.

Classic DH makes the mechanism visible.

X25519 shows what the modern streamlined version looks like.

Client-side handshake structure

Here is the current client handshake implementation:

def client_handshake(sock) -> bytes:
    """Perform the client side of the X25519 handshake.

    Returns the 32-byte shared secret.
    """
    print("\\n[handshake] Client: starting X25519 handshake")

    # Step 1: Generate an ephemeral X25519 keypair.
    # "Ephemeral" means we create a fresh keypair for this session only.
    # The private key never leaves this process and is discarded after use.
    client_private = X25519PrivateKey.generate()
    client_public = client_private.public_key()
    client_public_bytes = client_public.public_bytes(Encoding.Raw, PublicFormat.Raw)

    # Step 2: Send ClientHello with our public key.
    client_hello = encode_message(
        [
            (TAG_X25519_PUBLIC, client_public_bytes),
        ]
    )
    send_record(sock, client_hello)

    # Step 3: Receive ServerHello with the server's public key.
    server_hello_raw = recv_record(sock)
    fields = decode_message(server_hello_raw)
    server_public_bytes = None
    for tag, value in fields:
        if tag == TAG_X25519_PUBLIC:
            server_public_bytes = value
    if server_public_bytes is None:
        raise ValueError("ServerHello missing X25519 public key")

    # Deserialize the server's public key from raw bytes.
    server_public = X25519PublicKey.from_public_bytes(server_public_bytes)

    # Step 4: Compute the shared secret.
    # X25519(client_private, server_public) = X25519(server_private, client_public)
    # This is the elliptic-curve equivalent of g^(ab) mod p from v1.
    shared_secret = client_private.exchange(server_public)

    return shared_secret

I like this version because it makes the transition very clear.

The client code no longer has to think about p and g at all. It just performs the handshake, gets the shared secret, and prints it. That is exactly the point of this stage in the series: the workflow becomes smaller, but the underlying purpose stays the same.

What changed conceptually

Compared to the classic DH version, the protocol has become simpler in three important ways.

1. No explicit shared public parameters in the handshake

In the previous version, the client sent the prime and generator so the whole structure of classic Diffie-Hellman stayed visible.

Now that goes away.

X25519 already gives us a fixed, standard structure for the exchange, so the handshake only needs to carry the public key material.

That makes the protocol smaller and cleaner.

2. The public values are much more compact

In the classic DH version, the public values were tied to a large prime-field construction and looked much heavier in the protocol.

In this version, the public keys are just 32 bytes.

That is a huge practical simplification.

3. The code starts to look more like real modern protocol code

This line from the comments says it well:

generate(), exchange(), done.

That is exactly the feeling this section should create.

We are still doing public-key exchange.

We are still deriving a shared secret.

But the implementation shape is now much closer to what modern systems actually use.

What this version still does not solve

Even after switching to X25519, this version is still simplified:

• there is still no authentication

• the shared secret is not yet turned into session keys

• there is still no record-layer encryption using the new keys

In the next step, we will add HKDF and derive proper working session keys from it.

That is where the handshake starts to connect back to the record protection we built earlier.

Implementation Part 3 — Deriving Session Keys with HKDF

(The whole code can be find here: https://github.com/DmytroHuzz/rebuilding_tls/tree/main/part_3/v3_hkdf_session_keys)

At this point, both the classic Diffie-Hellman version and the X25519 version give us the same kind of output:

a shared secret that both sides can compute independently.

That is already a big step forward compared to the pre-shared-key model from the previous parts. The connection can now create fresh key material dynamically instead of starting with one hardcoded application key.

But there is still one important design question left:

should we use that raw shared secret directly as the application key?

For a toy demo, we probably could.

But even here, that would be the wrong direction.

Because a cleaner protocol separates these two ideas:

• the handshake creates a shared secret

• the protocol derives working session keys from that secret

That is exactly where HKDF comes in.

HKDF is a key-derivation function. Its job is not to invent secrecy out of nowhere, but to take existing secret material and turn it into keys that are better structured and easier to use safely inside the protocol.

So instead of treating the X25519 output as “the AES key,” we will use HKDF to derive proper session keys from it.

That already makes the protocol feel much closer to real TLS.

What changes conceptually

The structure now becomes:

X25519 shared secret
        |
        v
      HKDF
        |
        v
  session key material
        |
        v
 protected application data

This is an important shift.

Before this step, the handshake produced something secret and we could have stopped there.

After this step, the handshake produces an input to a key schedule.

That is a much better protocol design.

Why this matters

There are two main reasons to do this.

1. The raw shared secret is handshake output, not final protocol state

The shared secret is the result of key exchange. That does not automatically mean it should be used directly as the application-data key.

Protocols usually want a cleaner boundary:

• handshake result first

• working keys second

2. We can derive keys for different purposes

Once we introduce a key-derivation step, we are no longer forced into “one secret for everything.”

Even in this toy protocol, that opens the door to a much more realistic design.

For example, instead of one single AEAD key, we can derive:

• client → server key

• server → client key

That is already much closer to how real secure protocols think.

Deriving the keys

In the current implementation, HKDF takes the X25519 shared secret and stretches it into 64 bytes of key material.

Then that material is split into two 32-byte keys:

• one for traffic from client to server

• one for traffic from server to client

That gives us directional keys instead of one shared application key for both directions.

Here is the key schedule:

# key_schedule_x25519.py
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.hkdf import HKDF

def derive_session_keys(shared_secret: bytes) -> tuple[bytes, bytes]:
    key_material = HKDF(
        algorithm=hashes.SHA256(),
        length=64,
        salt=None,
        info=b"toy-tls-part-3-x25519",
    ).derive(shared_secret)

    client_to_server_key = key_material[:32]
    server_to_client_key = key_material[32:]

    return client_to_server_key, server_to_client_key

I like this step a lot because it is small in code, but it changes the protocol mindset in an important way.

We are no longer thinking:

handshake gives us the key

We are now thinking:

handshake gives us secret material, and the protocol derives the keys it actually wants to use

That is a much stronger model.

A small but important detail

Notice that the two sides must interpret the derived keys consistently.

If the client treats the first 32 bytes as the client → server key, then the server must do the same. Otherwise the channel will immediately break.

So now the handshake is not only producing shared secret material. It is also establishing a shared rule for how that material becomes working traffic keys.

That is another reason protocols need structure, not just primitives.

Connecting HKDF back to the record layer

Now we can finally connect this part back to what we built earlier.

In Part 2, we already built an AEAD-protected record layer. But that record layer still depended on hardcoded keys.

Now that changes.

The AEAD layer no longer starts with a static key from configuration.

It receives fresh traffic keys from the handshake.

So the protocol shape becomes:

Handshake -> X25519 shared secret -> HKDF -> directional session keys -> AEAD protected records

That is a major milestone in the series.

At this point, the protocol no longer just looks secure because we wrapped some bytes in encryption. It now has a real high-level structure:

• first establish shared key material

• then derive traffic keys

• then use those keys to protect application data

That is already much closer to the shape of real TLS.

Using the new session keys

Once the keys are derived, the record layer can use them directly.

Conceptually, the flow now looks like this:

Client

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client:
    client.connect((HOST, PORT))
    print(f"Connected to {HOST}:{PORT}")

    # ==========================================
    # PHASE 1: HANDSHAKE
    # ==========================================
    # New in Part 3: the handshake dynamically establishes session keys.
    # No pre-shared secret needed.
    client_write_key, server_write_key = client_handshake(client)

    # ==========================================
    # PHASE 2: APPLICATION DATA
    # ==========================================
    # The record layer now uses HKDF-derived keys instead of hardcoded ones.
    # The record format is the same as Part 2 Stage 3 (AEAD).

    # --- Send request (encrypted with client_write_key) ---
    protected = protect_record(client_write_key, send_seq, request)
    send_record(client, protected)
    send_seq += 1

    # --- Receive response (decrypted with server_write_key) ---
    raw_response = recv_record(client)

    try:
        response = unprotect_record(server_write_key, recv_seq, raw_response)
        recv_seq += 1
        print(f"\\n  Decrypted response:\\n  {response.decode('utf-8')}")
    except Exception as e:
        print(f"\\n  *** REJECTED: {e} ***")

print("\\nDone.")

• use client_write_key to protect outgoing application data

• use server_write_key to unprotect incoming application data

Server

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server:
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server.bind((HOST, PORT))
    server.listen(1)
    print(f"Listening on {HOST}:{PORT}")

    conn, addr = server.accept()
    with conn:
        # ==========================================
        # PHASE 1: HANDSHAKE
        # ==========================================
        client_write_key, server_write_key = server_handshake(conn)

        # ==========================================
        # PHASE 2: APPLICATION DATA
        # ==========================================

        # --- Receive request (decrypted with client_write_key) ---
        raw_request = recv_record(conn)

        try:
            request = unprotect_record(client_write_key, recv_seq, raw_request)
            recv_seq += 1
        except Exception as e:
            print(f"\\n  *** REJECTED: {e} ***")
            print("  Connection closed — refusing to process invalid data.")
        else:

            # --- Send response (encrypted with server_write_key) ---
            response = (
                "HTTP/1.1 200 OK\\r\\n"
                "Content-Type: text/plain\\r\\n"
                "Content-Length: 13\\r\\n\\r\\n"
                "hello, client"
            ).encode("utf-8")

            protected = protect_record(server_write_key, send_seq, response)
            send_record(conn, protected)
            send_seq += 1

print("\\nDone.")

• use client_write_key to unprotect incoming client traffic

• use server_write_key to protect outgoing server traffic

That means the two directions are now separated.

This is cleaner than one symmetric application key shared blindly by both directions, and it makes the protocol feel more deliberate.

Even in this simplified version, that is a meaningful step.

What this step really gave us

By adding HKDF, we improved the protocol in a way that is easy to underestimate.

We did not just “derive another key.”

We made the protocol architecture cleaner.

Now the handshake and the traffic layer are connected in a more principled way:

• the handshake creates shared secret material

• the key schedule turns that material into working keys

• the record layer consumes those keys

This is a much better model than treating the raw X25519 result as the final answer.

And it brings us one step closer to real TLS, where key derivation is not an optional detail, but one of the central pieces of the protocol design.

But we are still not secure

And now we arrive at the uncomfortable but necessary part.

Even with:

• a real handshake

• X25519

• HKDF

• fresh directional session keys

• AEAD-protected records

the protocol still cannot be considered secure enough.

Why?

Because all of this still says nothing about who is on the other side.

The handshake can successfully create shared secrets.

HKDF can successfully derive traffic keys.

The record layer can successfully protect application data.

And an attacker can still sit in the middle and run two separate handshakes.

That is the next lesson.

Still Not Secure — The Man-in-the-Middle Problem

At this point, our protocol already looks much more serious than the one we started with.

We now have:

• a real handshake

• fresh shared secrets

• X25519 instead of a pre-shared application key

• HKDF-derived session keys

• AEAD-protected application records

That is a long way from the fake secure channel in Part 1.

But it is still not enough.

The missing piece is one of the most important ideas in this whole series:

key exchange is not authentication

That sentence is easy to read quickly and move on from. But it is worth stopping here, because this is exactly where many protocols fail.

Our handshake proves that both sides can derive the same shared secret.

What it does not prove is:

who is actually on the other side.

And that difference is the whole problem.

The attack

Imagine an active attacker sitting between the client and the server.

Let’s call her Mallory.

The client thinks it is talking to the server.

The server thinks it is talking to the client.

But Mallory intercepts the handshake and replaces the exchanged public keys with her own.

In simplified form, the flow looks like this:

And now something very important happens.

The handshake still “works.”

But it works in the wrong way.

• the client ends up with a shared secret with Mallory

• the server ends up with a different shared secret with Mallory

• and Mallory now has one valid secure channel to each side

From the point of view of the client and the server, everything looks normal:

• key exchange succeeded

• keys were derived

• encrypted records verify correctly

• AEAD tags are valid

And yet the protocol has already failed.

Because Mallory can now:

1. decrypt the client’s traffic

2. read it or modify it

3. re-encrypt it toward the server

4. receive the server’s response

5. read it or modify it

6. re-encrypt it back toward the client

Neither side can detect this.

In The Next Article — Building the Certificate Infrastructure

The handshake only proves one thing:

“I computed a shared secret with whoever sent me this public key.”

It does not prove:

“This public key came from the server I actually intended to talk to.”

That is the missing half.

To fix this, the client needs a way to verify that the public key it receives during the handshake actually belongs to the server it wanted to talk to.

That is where the next layer enters:

• certificates

• signatures

• trust chains

• certificate authorities

In other words, this is where the protocol must stop proving only that “someone” is there and start proving who that someone is.

That is exactly what the next article will build.

Summary

Our protocol now has secrecy against passive observers.

It has integrity for protected records.

It has fresh session keys.

But it still does not have identity.

And without identity, a correct shared secret with the wrong party is still a protocol failure.

That is the deeper lesson of Part 3.

Part 1 taught us:

confidentiality is not integrity

Part 2 taught us:

protecting records is not the same thing as establishing trust

And now Part 3 adds the next lesson:

key exchange is not authentication

That we will solve in the next article!

Final Code

The full code for this part is available here:

GitHub: https://github.com/DmytroHuzz/rebuilding_tls/tree/main/part_3

Next article

In the first part of this series, we built our first fake secure channel:

We took a simple socket-based client and server, wrapped their communication in AES-CTR with a shared secret key, and got something that already looked much more serious than plain TCP. The traffic stopped being transparent. A passive observer could no longer read the request and response directly.

That was real progress.

But it still had a fatal flaw.

The receiver had no way to know whether the encrypted message had been changed on the way.

Encryption hid the bytes.

It did not protect their meaning.

So in this part, we will fix that.

We will first add a MAC so the receiver can detect tampering. Then we will make the record layer a little less naive by adding a sequence number. And after that, we will take one more step toward the real world and move to AEAD, because that is how modern secure protocols usually protect records.

We still will not have real TLS when we are done.

But we will have a much more serious record layer than the one from Part 1.

What we will build in this part

The plan for this article is simple:

• briefly introduce MACs

• add HMAC to our encrypted record format

• make tampering detectable

• add a sequence number to each record

• explain why sequence numbers matter

• then move from our hand-built “encrypt + MAC” construction to AEAD, because that is the approach real systems usually use

Just like in Part 1, I want to keep the pattern simple:

• explain the idea

• show the code

• explain what changed

• explain what is still broken

Subscribe now

Why encryption still was not enough

At the end of Part 1, our protocol already had one real property:

• confidentiality against passive observers

That mattered.

But it still failed against active attackers.

Because AES-CTR by itself does not provide integrity, an attacker could modify ciphertext and the receiver would still decrypt it and trust the result. That was the main lesson of the first article:

confidentiality is not integrity

So the next missing property is obvious.

The receiver needs a way to verify that the message arrived unchanged.

That is what a MAC gives us.

A very short note on MACs

MAC stands for Message Authentication Code.

Very roughly, it is a cryptographic tag computed over a message using a secret key.

The sender computes the tag and sends it together with the message.

The receiver recomputes the tag and compares it with the one that was received.

If the tags match, the receiver can trust that:

• the message was not modified

• and it was created by someone who knows the MAC key

If the tags do not match, the message must be rejected.

In this article, we will use HMAC-SHA256.

I do not want to go too deep into HMAC itself here, because the goal of this series is to understand TLS as a protocol. But if you want a deeper explanation of MACs and HMAC, I already wrote about them in my cryptography series, and I’ll link that here.

So for our purposes, the important idea is simple:

encryption hides the message

MAC protects the message from silent modification

That is the missing half we need.

Adding HMAC to the channel

Let’s start by upgrading the record format from Part 1.

In Part 1, our protected payload was basically:

nonce || ciphertext

Now we will add a MAC tag:

nonce || ciphertext || tag

And the sender will compute the HMAC over:

nonce || ciphertext

So the full logic becomes:

Sender

1. encrypt plaintext with AES-CTR

2. compute HMAC over nonce || ciphertext

3. send nonce || ciphertext || tag

Receiver

1. read nonce || ciphertext || tag

2. recompute HMAC over nonce || ciphertext

3. compare tags

4. only if they match, decrypt the ciphertext

5. otherwise reject the message

There is one more small improvement I want to make here.

Instead of using one key for everything, we will already separate them:

• one key for encryption

• one key for HMAC

This is still a toy setup, but it is better design than reusing the same bytes for every cryptographic job.

HMAC helpers

import os
import hmac
import hashlib

from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes

# ---------------------------------------------------------------------------
# Keys — hardcoded for educational purposes.
# In a real protocol, these would be derived from a key exchange (e.g.,
# Diffie-Hellman), not embedded in source code.
# ---------------------------------------------------------------------------

# 32-byte (256-bit) key for AES-256-CTR encryption.
ENC_KEY = b"0123456789ABCDEF0123456789ABCDEF"

# 32-byte key for HMAC-SHA256.  Separate from the encryption key.
MAC_KEY = b"HMAC_KEY_FOR_PART2_DEMO_1234567"

# HMAC-SHA256 produces a 32-byte (256-bit) tag.
TAG_LEN = 32

# AES-CTR nonce is 16 bytes (128 bits).
NONCE_LEN = 16

def encrypt_then_mac(plaintext: bytes) -> bytes:
    """Encrypt a plaintext and append an HMAC tag.

    Returns: nonce (16 B) || ciphertext (N B) || tag (32 B)
    """

    # Step 1: Generate a fresh random nonce for AES-CTR.
    # A new nonce MUST be used for every record — reusing a nonce with
    # the same key completely breaks CTR-mode security.
    nonce = os.urandom(NONCE_LEN)

    # Step 2: Encrypt the plaintext with AES-256-CTR.
    cipher = Cipher(algorithms.AES(ENC_KEY), modes.CTR(nonce))
    encryptor = cipher.encryptor()
    ciphertext = encryptor.update(plaintext) + encryptor.finalize()

    # Step 3: Compute HMAC-SHA256 over (nonce || ciphertext).
    # New in Part 2: we authenticate the encrypted record before sending it.
    # The HMAC input includes the nonce so an attacker cannot swap nonces
    # between records without detection.
    mac_input = nonce + ciphertext
    tag = hmac.new(MAC_KEY, mac_input, hashlib.sha256).digest()

    print(f"  [crypto_hmac] encrypt_then_mac:")
    print(f"    nonce    = {nonce.hex()[:32]}...")
    print(f"    ct_len   = {len(ciphertext)} bytes")
    print(f"    tag      = {tag.hex()[:32]}...")

    # Step 4: Assemble the wire format.
    return nonce + ciphertext + tag

def verify_then_decrypt(payload: bytes) -> bytes:
    """Verify the HMAC tag, then decrypt if valid.

    Expects: nonce (16 B) || ciphertext (N B) || tag (32 B)
    Raises ValueError if the tag does not match.
    """

    # Step 1: Parse the record into its components.
    # The tag is always the last 32 bytes.  The nonce is the first 16.
    # Everything in between is ciphertext.
    if len(payload) < NONCE_LEN + TAG_LEN:
        raise ValueError("Record too short to contain nonce + tag")

    nonce = payload[:NONCE_LEN]
    ciphertext = payload[NONCE_LEN:-TAG_LEN]
    received_tag = payload[-TAG_LEN:]

    # Step 2: Recompute the HMAC over (nonce || ciphertext).
    mac_input = nonce + ciphertext
    expected_tag = hmac.new(MAC_KEY, mac_input, hashlib.sha256).digest()

    # Step 3: Compare tags using constant-time comparison.
    # hmac.compare_digest() prevents timing side-channel attacks.
    # A naive `==` comparison can leak information about which byte
    # position differs first, allowing an attacker to forge a valid
    # tag byte by byte.
    if not hmac.compare_digest(received_tag, expected_tag):
        print("  [crypto_hmac] *** MAC VERIFICATION FAILED — record rejected ***")
        raise ValueError("HMAC verification failed — record has been tampered with")

    print("  [crypto_hmac] MAC verification: OK")

    # Step 4: Decrypt only after verification succeeds.
    # This is the key benefit of encrypt-then-MAC: we never process
    # unauthenticated ciphertext.
    cipher = Cipher(algorithms.AES(ENC_KEY), modes.CTR(nonce))
    decryptor = cipher.decryptor()
    plaintext = decryptor.update(ciphertext) + decryptor.finalize()

    return plaintext

This is the first big improvement over Part 1.

The important change is not just that we added a tag.

It is that the receiver no longer blindly trusts ciphertext and only then discovers what it means. Now the receiver first checks whether the record is authentic and unchanged.

That is a very different protocol posture.

Updating the client and server

Now let’s plug this into the channel.

HMAC-based client

import socket

from framing import send_record, recv_record
from crypto_hmac import encrypt_then_mac, verify_then_decrypt

HOST = "127.0.0.1"
PORT = 9001

# A toy HTTP-like request — same spirit as Part 1.
request = (
    "GET /transfer?to=bob&amount=100 HTTP/1.1\\r\\nHost: localhost\\r\\n\\r\\n"
).encode("utf-8")

print("=" * 60)
print("Part 2 — HMAC Client (encrypt-then-MAC)")
print("=" * 60)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client:
    client.connect((HOST, PORT))
    print(f"Connected to {HOST}:{PORT}")

    # ----- SEND REQUEST -----
    print("\\n--- Sending request ---")
    protected = encrypt_then_mac(request)
    send_record(client, protected)
    print(f"  Record sent ({len(protected)} bytes on wire)")

    # ----- RECEIVE RESPONSE -----
    print("\\n--- Receiving response ---")
    raw_response = recv_record(client)
    response = verify_then_decrypt(raw_response)
    print(f"\\n  Decrypted response:\\n  {response.decode('utf-8')}")

print("\\nDone.")

HMAC-based server

import socket

from framing import send_record, recv_record
from crypto_hmac import encrypt_then_mac, verify_then_decrypt

HOST = "127.0.0.1"
PORT = 9001

print("=" * 60)
print("Part 2 — HMAC Server (encrypt-then-MAC)")
print("=" * 60)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server:
    # SO_REUSEADDR lets us restart the server immediately without waiting
    # for the OS to release the port from TIME_WAIT state.
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server.bind((HOST, PORT))
    server.listen(1)
    print(f"Listening on {HOST}:{PORT}")

    conn, addr = server.accept()
    with conn:
        print(f"Connected by {addr}")

        # ----- RECEIVE REQUEST -----
        print("\\n--- Receiving request ---")
        raw_request = recv_record(conn)

        try:
            request = verify_then_decrypt(raw_request)
        except ValueError as e:
            # New in Part 2: if the MAC fails, we reject the record loudly.
            # In Part 1 we had no way to detect tampering at all.
            print(f"\\n  *** REJECTED: {e} ***")
            print("  Connection closed — refusing to process tampered data.")
        else:
            print(f"\\n  Decrypted request:\\n  {request.decode('utf-8')}")

            # ----- SEND RESPONSE -----
            print("--- Sending response ---")
            response = (
                "HTTP/1.1 200 OK\\r\\n"
                "Content-Type: text/plain\\r\\n"
                "Content-Length: 13\\r\\n\\r\\n"
                "hello, client"
            ).encode("utf-8")

            protected = encrypt_then_mac(response)
            send_record(conn, protected)
            print(f"  Record sent ({len(protected)} bytes on wire)")

print("\\nDone.")

The shape of the channel is still familiar.

That matters.

We did not replace the whole design.

We strengthened one missing property.

That is how protocol evolution should feel.

Let’s check it on the wire.

We try to start the server and client, which we just created.

Here is our client request’s data.

-- Sending request ---
[crypto_hmac] encrypt_then_mac:
nonce = 387f0065f8915133473597d8cef15f34...
ct_len = 61 bytes
tag = b7f0db369e5d2a221a18f2d167b5b3a8...
Record sent (109 bytes on wire)

Detecting tampering

Now let’s revisit the failure from Part 1.

Previously, if someone modified the ciphertext, the receiver would still decrypt it and accept modified plaintext.

Now that should no longer work.

Here is a tiny tampering demo:

# tampering_demo_hmac.py
from crypto_hmac import encrypt_then_mac, verify_then_decrypt

original = b"amount=100"
protected = encrypt_then_mac(original)

tampered = bytearray(protected)
tampered[20] ^= 0x08  # flip one bit somewhere in the encrypted body

try:
    result = verify_then_decrypt(bytes(tampered))
    print("Unexpected success:", result)
except ValueError as e:
    print("Tampering detected:", e)

Now the result should be rejection, not silent acceptance.

That is exactly what we wanted.

This is the moment where our channel stops being merely “encrypted” and starts being “protected.”

Because now the receiver does not just recover bytes. It verifies them first.

That is a serious step.

Why we also need a sequence number

At this point, we fixed the big flaw from Part 1: silent tampering.

But the record layer is still naive.

Why?

Because even with a valid HMAC, the receiver still has no sense of record position or freshness.

Imagine an attacker records one valid protected message and sends it again later.

The HMAC is still valid.

The ciphertext is still valid.

And unless the receiver keeps some state, it may accept the same record again.

That means integrity alone is not the whole story.

We also need some sense of:

• order

• position

• repetition

• replay

This is where sequence numbers come in.

A sequence number is just a counter that increases with every record:

• first record = 0

• next = 1

• next = 2

• and so on

We then include that sequence number in the authenticated data, so the receiver does not just verify “these bytes were protected,” but also “these bytes belong in this position in the stream.”

That makes the record layer much less naive.

It still does not solve every replay problem in every possible system. But for our toy protocol, it is a very good next step.

Updating the record format

Now our record becomes:

seq || nonce || ciphertext || tag

And our HMAC input becomes:

seq || nonce || ciphertext

So the sender and receiver now both need a little bit of state:

• the sender tracks the next sequence number to send

• the receiver tracks the next sequence number it expects

This is one of those moments where secure transport starts looking more like a real protocol and less like “some crypto around a socket.”

Sequence-aware HMAC helper

# crypto_hmac_seq.py
import os
import hmac
import hashlib
import struct

from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes

# ---------------------------------------------------------------------------
# Keys — same as crypto_hmac.py, hardcoded for education.
# ---------------------------------------------------------------------------
ENC_KEY = b"0123456789ABCDEF0123456789ABCDEF"
MAC_KEY = b"HMAC_KEY_FOR_PART2_DEMO_1234567"

TAG_LEN = 32  # HMAC-SHA256 output: 32 bytes (256 bits)
NONCE_LEN = 16  # AES-CTR nonce: 16 bytes (128 bits)
SEQ_LEN = 8  # Sequence number: 8 bytes (64-bit unsigned integer)

def protect_record(seq: int, plaintext: bytes) -> bytes:
    """Encrypt a plaintext record and attach a sequence-aware HMAC tag.

    Args:
        seq:       The current send-side sequence number (0, 1, 2, …).
        plaintext: The message to protect.

    Returns:
        seq (8 B) || nonce (16 B) || ciphertext (N B) || tag (32 B)
    """

    # Pack the sequence number as an 8-byte big-endian unsigned integer.
    # "!Q" = network byte order, unsigned 64-bit.
    seq_bytes = struct.pack("!Q", seq)

    # Generate a fresh AES-CTR nonce.
    nonce = os.urandom(NONCE_LEN)

    # Encrypt the plaintext.
    cipher = Cipher(algorithms.AES(ENC_KEY), modes.CTR(nonce))
    encryptor = cipher.encryptor()
    ciphertext = encryptor.update(plaintext) + encryptor.finalize()

    # Compute HMAC over (seq || nonce || ciphertext).
    # The sequence number is included in the MAC input so the integrity
    # check also covers record order/position in the stream.
    mac_input = seq_bytes + nonce + ciphertext
    tag = hmac.new(MAC_KEY, mac_input, hashlib.sha256).digest()

    return seq_bytes + nonce + ciphertext + tag

def verify_and_unprotect(expected_seq: int, payload: bytes) -> bytes:
    """Verify the HMAC and sequence number, then decrypt.

    Args:
        expected_seq: The sequence number the receiver expects next.
        payload:      The raw bytes received: seq || nonce || ct || tag.

    Returns:
        The decrypted plaintext.

    Raises:
        ValueError if the MAC is invalid or the sequence number is wrong.
    """

    min_len = SEQ_LEN + NONCE_LEN + TAG_LEN
    if len(payload) < min_len:
        raise ValueError("Record too short")

    # Step 1: Parse the record.
    seq_bytes = payload[:SEQ_LEN]
    nonce = payload[SEQ_LEN : SEQ_LEN + NONCE_LEN]
    ciphertext = payload[SEQ_LEN + NONCE_LEN : -TAG_LEN]
    received_tag = payload[-TAG_LEN:]

    # Step 2: Recompute HMAC over (seq || nonce || ciphertext).
    mac_input = seq_bytes + nonce + ciphertext
    expected_tag = hmac.new(MAC_KEY, mac_input, hashlib.sha256).digest()

    # Step 3: Constant-time tag comparison.
    if not hmac.compare_digest(received_tag, expected_tag):
        print("  [crypto_hmac_seq] *** MAC VERIFICATION FAILED ***")
        raise ValueError("HMAC verification failed — record tampered or replayed")

    print("  [crypto_hmac_seq] MAC verification: OK")

    # Step 4: Check the sequence number matches what we expect.
    # Even though the MAC already covers the sequence number (so an
    # attacker cannot change it without invalidating the MAC), we still
    # explicitly verify that it matches our counter.  This catches
    # replayed or reordered records that carry a valid MAC but belong
    # to a different position in the stream.
    (received_seq,) = struct.unpack("!Q", seq_bytes)
    if received_seq != expected_seq:
        print(
            f"  [crypto_hmac_seq] *** SEQUENCE MISMATCH: "
            f"got {received_seq}, expected {expected_seq} ***"
        )
        raise ValueError(
            f"Sequence number mismatch: got {received_seq}, expected {expected_seq}"
        )

    print(
        f"  [crypto_hmac_seq] Sequence number: {received_seq} (expected {expected_seq}) — OK"
    )

    # Step 5: Decrypt.
    cipher = Cipher(algorithms.AES(ENC_KEY), modes.CTR(nonce))
    decryptor = cipher.decryptor()
    plaintext = decryptor.update(ciphertext) + decryptor.finalize()

    return plaintext

And now the channel has a bit more memory.

Not just “is this record authentic?”

But also “is this the record I expected next?”

That is a real protocol improvement.

Updating the client and server

Now let’s plug this into the channel.

Sequence-aware HMAC-based client

# client_v2_hmac_seq.py
import socket

from framing import send_record, recv_record
from crypto_hmac_seq import protect_record, verify_and_unprotect

HOST = "127.0.0.1"
PORT = 9003

# Sequence counters — sender and receiver each maintain their own.
# The sender increments after each record sent.
# The receiver expects consecutive values starting from 0.
send_seq = 0
recv_seq = 0

# A toy HTTP-like request — same spirit as Part 1.
request = (
    "GET /transfer?to=bob&amount=100 HTTP/1.1\\r\\nHost: localhost\\r\\n\\r\\n"
).encode("utf-8")

print("=" * 60)
print("Part 2 — HMAC + Sequence Numbers Client (Stage 2)")
print("=" * 60)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client:
    client.connect((HOST, PORT))
    print(f"Connected to {HOST}:{PORT}")

    # ----- SEND REQUEST -----
    print(f"\\n--- Sending request (send_seq={send_seq}) ---")
    protected = protect_record(send_seq, request)
    send_record(client, protected)
    send_seq += 1
    print(f"  Record sent ({len(protected)} bytes on wire)")

    # ----- RECEIVE RESPONSE -----
    print(f"\\n--- Receiving response (expecting recv_seq={recv_seq}) ---")
    raw_response = recv_record(client)

    try:
        response = verify_and_unprotect(recv_seq, raw_response)
        recv_seq += 1
        print(f"\\n  Decrypted response:\\n  {response.decode('utf-8')}")
    except ValueError as e:
        print(f"\\n  *** REJECTED: {e} ***")

print("\\nDone.")

Sequence-aware HMAC-based server

# server_v2_hmac_seq.py
import socket

from framing import send_record, recv_record
from crypto_hmac_seq import protect_record, verify_and_unprotect

HOST = "127.0.0.1"
PORT = 9003

# Sequence counters.
# The server's recv_seq tracks the client's send_seq, and vice versa.
send_seq = 0
recv_seq = 0

print("=" * 60)
print("Part 2 — HMAC + Sequence Numbers Server (Stage 2)")
print("=" * 60)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server:
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server.bind((HOST, PORT))
    server.listen(1)
    print(f"Listening on {HOST}:{PORT}")

    conn, addr = server.accept()
    with conn:
        print(f"Connected by {addr}")

        # ----- RECEIVE REQUEST -----
        print(f"\\n--- Receiving request (expecting recv_seq={recv_seq}) ---")
        raw_request = recv_record(conn)

        try:
            request = verify_and_unprotect(recv_seq, raw_request)
            recv_seq += 1
        except ValueError as e:
            # Rejection: either the MAC is invalid, the sequence number
            # is wrong, or the data was tampered with / replayed.
            print(f"\\n  *** REJECTED: {e} ***")
            print("  Connection closed — refusing to process invalid data.")
        else:
            print(f"\\n  Decrypted request:\\n  {request.decode('utf-8')}")

            # ----- SEND RESPONSE -----
            print(f"--- Sending response (send_seq={send_seq}) ---")
            response = (
                "HTTP/1.1 200 OK\\r\\n"
                "Content-Type: text/plain\\r\\n"
                "Content-Length: 13\\r\\n\\r\\n"
                "hello, client"
            ).encode("utf-8")

            protected = protect_record(send_seq, response)
            send_record(conn, protected)
            send_seq += 1
            print(f"  Record sent ({len(protected)} bytes on wire)")

print("\\nDone.")

Why real-world systems usually do not stop here

At this point, we have something much stronger than Part 1.

We have:

• encryption

• integrity protection

• message authentication

• sequence-aware records

That is already a meaningful protocol.

But if you look at how real systems are usually built, they do not normally stop at manually composing:

• AES-CTR

• HMAC-SHA256

• explicit sequence-aware record protection

Why?

Because modern systems usually prefer a single primitive that gives confidentiality and integrity together.

That is where AEAD comes in.

We separated these properties on purpose because it makes the protocol easier to understand.

But the real world usually packages them together.

A very short note on AEAD

AEAD stands for Authenticated Encryption with Associated Data.

That sounds heavier than it really is.

The practical idea is simple:

An AEAD construction gives us:

• encryption

• integrity/authentication of the encrypted message

• and the ability to authenticate extra metadata that should not be encrypted

Common examples are:

• AES-GCM

• ChaCha20-Poly1305

This is much closer to how modern secure protocols protect records.

It is also why I wanted to include AEAD in this part. If we stopped only at “encrypt + HMAC,” we would understand the missing property better, but we would still be one step away from how modern systems actually package it.

So now we take that final step.

Moving our channel to AEAD

For the AEAD version, I will use AES-GCM.

The high-level idea is:

• the plaintext gets encrypted

• integrity/authentication is built in

• and we can include extra metadata as associated data

In our case, the sequence number is a good example of associated data.

That means:

• it does not need to be encrypted

• but it should still be authenticated

AEAD-based helper

# crypto_aead.py
import os
import struct

from cryptography.hazmat.primitives.ciphers.aead import AESGCM

# ---------------------------------------------------------------------------
# Key — a single 256-bit key for AES-GCM.
# With AEAD, we do NOT need separate encryption and MAC keys — the
# algorithm handles both internally.
# ---------------------------------------------------------------------------
AEAD_KEY = b"AEAD_KEY_PART2_DEMO_FOR_AES_GCM!"  # 32 bytes → AES-256-GCM

# AES-GCM nonce length: 12 bytes is the recommended (and most efficient) size.
NONCE_LEN = 12

# Sequence number: 8 bytes (64-bit unsigned integer), same as Stage 2.
SEQ_LEN = 8

def protect_record_aead(seq: int, plaintext: bytes) -> bytes:
    """Seal a plaintext record with AES-GCM.

    Args:
        seq:       The current send-side sequence number.
        plaintext: The message to protect.

    Returns:
        seq (8 B) || nonce (12 B) || ciphertext_and_tag (N+16 B)
    """

    # Pack the sequence number as associated data.
    # The sequence number is authenticated but sent in the clear — the
    # receiver needs it to know which counter value to expect.
    seq_bytes = struct.pack("!Q", seq)

    # Generate a random 12-byte nonce for AES-GCM.
    nonce = os.urandom(NONCE_LEN)

    # Create an AESGCM instance with our key.
    aesgcm = AESGCM(AEAD_KEY)

    # Encrypt and authenticate in one call.
    # AESGCM.encrypt(nonce, data, associated_data) returns
    # ciphertext || 16-byte authentication tag as a single bytes object.
    # The associated_data (seq_bytes) is authenticated but NOT encrypted.
    ciphertext_and_tag = aesgcm.encrypt(nonce, plaintext, seq_bytes)

    print(f"  [crypto_aead] protect_record_aead:")
    print(f"    seq      = {seq}")
    print(f"    nonce    = {nonce.hex()}")
    print(
        f"    sealed   = {len(ciphertext_and_tag)} bytes "
        f"(plaintext {len(plaintext)} + tag 16)"
    )

    return seq_bytes + nonce + ciphertext_and_tag

def unprotect_record_aead(expected_seq: int, payload: bytes) -> bytes:
    """Verify and decrypt an AES-GCM sealed record.

    Args:
        expected_seq: The sequence number the receiver expects next.
        payload:      seq (8 B) || nonce (12 B) || ciphertext_and_tag.

    Returns:
        The decrypted plaintext.

    Raises:
        ValueError if the sequence number is wrong.
        cryptography.exceptions.InvalidTag if decryption/auth fails.
    """

    min_len = SEQ_LEN + NONCE_LEN + 16  # at least seq + nonce + tag
    if len(payload) < min_len:
        raise ValueError("Record too short")

    # Step 1: Parse the record.
    seq_bytes = payload[:SEQ_LEN]
    nonce = payload[SEQ_LEN : SEQ_LEN + NONCE_LEN]
    ciphertext_and_tag = payload[SEQ_LEN + NONCE_LEN :]

    # Step 2: Check the sequence number.
    (received_seq,) = struct.unpack("!Q", seq_bytes)
    if received_seq != expected_seq:
        print(
            f"  [crypto_aead] *** SEQUENCE MISMATCH: "
            f"got {received_seq}, expected {expected_seq} ***"
        )
        raise ValueError(
            f"Sequence number mismatch: got {received_seq}, expected {expected_seq}"
        )

    print(
        f"  [crypto_aead] Sequence number: {received_seq} "
        f"(expected {expected_seq}) — OK"
    )

    # Step 3: Decrypt and verify in one call.
    # AESGCM.decrypt(nonce, data, associated_data) verifies the auth tag
    # and decrypts.  If anything was tampered with — the ciphertext, the
    # tag, or the associated data — it raises InvalidTag.
    aesgcm = AESGCM(AEAD_KEY)
    plaintext = aesgcm.decrypt(nonce, ciphertext_and_tag, seq_bytes)

    print(f"  [crypto_aead] AEAD decryption: OK ({len(plaintext)} bytes)")

    return plaintext

This code is noticeably simpler.

That is one of the big practical advantages of AEAD.

Instead of manually:

• encrypting

• computing HMAC

• verifying HMAC

• then decrypting

we use one primitive that already combines confidentiality and integrity.

And the sequence number fits naturally as associated data.

AEAD-based client

# client_v2_aead.py
import socket

from framing import send_record, recv_record
from crypto_aead import protect_record_aead, unprotect_record_aead

HOST = "127.0.0.1"
PORT = 9002

# Sequence counters — sender and receiver each maintain their own.
# The sender increments after each record sent.
# The receiver expects consecutive values starting from 0.
send_seq = 0
recv_seq = 0

# A toy HTTP-like request.
request = (
    "GET /transfer?to=bob&amount=100 HTTP/1.1\\r\\nHost: localhost\\r\\n\\r\\n"
).encode("utf-8")

print("=" * 60)
print("Part 2 — AEAD Client (AES-GCM)")
print("=" * 60)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client:
    client.connect((HOST, PORT))
    print(f"Connected to {HOST}:{PORT}")

    # ----- SEND REQUEST -----
    print(f"\\n--- Sending request (send_seq={send_seq}) ---")
    protected = protect_record_aead(send_seq, request)
    send_record(client, protected)
    send_seq += 1
    print(f"  Record sent ({len(protected)} bytes on wire)")

    # ----- RECEIVE RESPONSE -----
    print(f"\\n--- Receiving response (expecting recv_seq={recv_seq}) ---")
    raw_response = recv_record(client)

    try:
        response = unprotect_record_aead(recv_seq, raw_response)
        recv_seq += 1
        print(f"\\n  Decrypted response:\\n  {response.decode('utf-8')}")
    except Exception as e:
        print(f"\\n  *** REJECTED: {e} ***")

print("\\nDone.")

AEAD-based server

# server_v2_aead.py
import socket

from framing import send_record, recv_record
from crypto_aead import protect_record_aead, unprotect_record_aead

HOST = "127.0.0.1"
PORT = 9002

# Sequence counters.
# The server's recv_seq tracks the client's send_seq, and vice versa.
send_seq = 0
recv_seq = 0

print("=" * 60)
print("Part 2 — AEAD Server (AES-GCM)")
print("=" * 60)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server:
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server.bind((HOST, PORT))
    server.listen(1)
    print(f"Listening on {HOST}:{PORT}")

    conn, addr = server.accept()
    with conn:
        print(f"Connected by {addr}")

        # ----- RECEIVE REQUEST -----
        print(f"\\n--- Receiving request (expecting recv_seq={recv_seq}) ---")
        raw_request = recv_record(conn)

        try:
            request = unprotect_record_aead(recv_seq, raw_request)
            recv_seq += 1
        except Exception as e:
            # AEAD rejection: either the auth tag is invalid, the sequence
            # number is wrong, or the data was tampered with.
            print(f"\\n  *** REJECTED: {e} ***")
            print("  Connection closed — refusing to process invalid data.")
        else:
            print(f"\\n  Decrypted request:\\n  {request.decode('utf-8')}")

            # ----- SEND RESPONSE -----
            print(f"--- Sending response (send_seq={send_seq}) ---")
            response = (
                "HTTP/1.1 200 OK\\r\\n"
                "Content-Type: text/plain\\r\\n"
                "Content-Length: 13\\r\\n\\r\\n"
                "hello, client"
            ).encode("utf-8")

            protected = protect_record_aead(send_seq, response)
            send_record(conn, protected)
            send_seq += 1
            print(f"  Record sent ({len(protected)} bytes on wire)")

print("\\nDone.")

This version is already much closer to how modern secure transport actually protects records.

Not identical to TLS, of course. But structurally much closer.

What we gained

At this point, our channel is much stronger than the one from Part 1.

We now have:

• confidentiality

• integrity protection

• authenticated records

• sequence-aware message handling

• a much more realistic record protection design through AEAD

That is a big improvement.

The receiver is no longer just decrypting whatever arrives and trusting the result. Now the receiver can reject modified or structurally unexpected records.

That is a real protocol boundary.

What is still broken

And yet, even now, we are still very far from real TLS.

Because the biggest assumption in our design is still untouched:

both sides already share the necessary secret keys

That means we still do not know how to solve the next real problem:

• how do two strangers establish fresh secrets?

• how does the client know it is talking to the right server?

• how do we scale beyond hardcoded shared secrets?

• how do we build trust instead of assuming it?

We improved record protection a lot.

But we still do not have a real way to establish trust.

That is the next wall.

Summary

In this part, we took the encrypted but still incomplete channel from Part 1 and made it much more serious.

First, we added HMAC, which gave the receiver a way to detect tampering.

Then, we added a sequence number, which made the record layer less naive and bound records to their place in the stream.

Finally, we moved to AEAD, because in real-world systems confidentiality and integrity are usually protected together, not assembled manually from separate pieces.

So this article had two goals:

• understand the missing property explicitly

• then move toward the real-world shape of the solution

That is why we did not stop at HMAC.

But even after all of this, we still depend on one assumption that makes the whole thing unrealistic:

we are still starting with pre-shared secret keys.

And that is exactly what the next part will attack.

Next part — getting rid of the pre-shared key

So we stop here.

We now have a much better record layer than in Part 1. But we are still relying on a hardcoded shared secret, and that is not how real secure communication between strangers on the internet works.

In the next part, we will stop assuming both sides already share a secret.

We will start building a real handshake and establish fresh session keys instead.

That still will not give us full TLS.

But it will take us much closer to the real shape of the protocol.

Final code

I’ll put the full code for this part on GitHub here:

[GitHub link to final code]

Next article:

A year ago I wrote a series about how a web server works.

I started from a very primitive version and step by step moved toward the same core ideas modern production servers rely on. When I finished that series, I thought the next step would be small.

Wrap it in TLS. Make the communication secure.

It did not stay small for long.

What looked like a thin security layer on top of an existing server turned into a much deeper journey into cryptography, authentication, trust, certificates, protocol design, and many details usually hidden behind one familiar phrase: secure connection.

So this series is my attempt to approach TLS the same way I approached the web server: not as a finished black box, but as something we can rebuild from simpler pieces until its shape starts to make sense.

In this first part, we will start with the most naive version of the problem.

We will build a very simple socket-based communication channel, see that it is fully transparent, wrap it in encryption with a shared secret key, and then see why that is still not enough.

That will give us our first fake secure channel.

And that is exactly where we should start.

What we will build in this part

The plan for this article is simple:

• build a tiny socket-based client and server

• send plain text between them

• look at the traffic and see that everything is visible

• add shared-key encryption with AES-CTR

• make the traffic unreadable

• then show why encryption alone still does not give us a trustworthy secure protocol

We are not trying to build real TLS yet.

We are trying to make the first mistake on purpose.

Because once that mistake becomes visible, the next piece of the protocol stops looking optional.

TLS is not SSL

Before we start, one small clarification.

People still often say “SSL” when they talk about secure communication on the web. But SSL is the older family of protocols. TLS is its successor.

So when people say things like “SSL certificate” or “SSL connection,” in practice they usually mean TLS.

For modern systems, the relevant protocols are TLS, especially TLS 1.2 and TLS 1.3. This series is about understanding the ideas behind TLS by rebuilding simpler versions of the problems it solves.

And instead of starting from the finished protocol, we will begin one layer lower — with plain socket communication.

Step 1 — A plain socket-based communication channel

Let’s start with the smallest possible thing: a tiny TCP server and a tiny TCP client.

The client will send an HTTP-like request.

The server will read it and return an HTTP-like response.

Nothing secure yet. Just raw bytes moving over a socket.

Plain server

# server_plain.py
import socket

HOST = "127.0.0.1"
PORT = 8081

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server:
    server.bind((HOST, PORT))
    server.listen(1)

    print(f"Listening on {HOST}:{PORT}")
    conn, addr = server.accept()

    with conn:
        print(f"Connected by {addr}")

        data = conn.recv(4096)
        request = data.decode("utf-8")
        print("Received request:")
        print(request)

        response = (
            "HTTP/1.1 200 OK\\r\\n"
            "Content-Type: text/plain\\r\\n"
            "Content-Length: 13\\r\\n"
            "\\r\\n"
            "hello, client"
        )
        conn.sendall(response.encode("utf-8"))

Plain client

# client_plain.py
import socket

HOST = "127.0.0.1"
PORT = 8081

request = (
    "GET /transfer?to=bob&amount=100 HTTP/1.1\\r\\n"
    "Host: localhost\\r\\n"
    "\\r\\n"
)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client:
    client.connect((HOST, PORT))
    client.sendall(request.encode("utf-8"))

    response = client.recv(4096)
    print("Received response:")
    print(response.decode("utf-8"))

This is intentionally tiny.

The client sends a request like this:

GET /transfer?to=bob&amount=100 HTTP/1.1
Host: localhost

The server reads it and sends a response back.

That is all.

And because it is all plain TCP, anyone who can observe the traffic can read it directly.

Looking at the traffic

If you capture this communication in Wireshark, the request and response are fully visible in clear text.

That is our baseline.

The client can read it.

The server can read it.

And anyone on the wire can read it too.

So the first obvious idea is also the first naive one:

If the problem is that everyone can read the bytes, let’s encrypt the bytes.

That sounds reasonable.

And it is still not enough.

Step 2 — Turning bytes into records

Before we add encryption, we need one small but important thing: structure.

TCP gives us a byte stream.

It does not give us message boundaries.

So once we stop sending plain text directly and start sending encrypted blobs, we need a way to tell the receiver how many bytes belong to one logical message.

That means even before security, we need a little bit of protocol design.

Let’s define the smallest possible record format:

• 4 bytes: payload length

• N bytes: payload

+----------+-----------+
|  length  |  payload  |
| (4 bytes)|  (varies) |
+----------+-----------+

That is enough for our first version.

# framing.py
import struct

def send_record(sock, payload: bytes) -> None:
    header = struct.pack("!I", len(payload))
    sock.sendall(header + payload)

def recv_exact(sock, n: int) -> bytes:
    chunks = []
    remaining = n

    while remaining > 0:
        chunk = sock.recv(remaining)
        if not chunk:
            raise ConnectionError("Connection closed while reading data")
        chunks.append(chunk)
        remaining -= len(chunk)

    return b"".join(chunks)

def recv_record(sock) -> bytes:
    header = recv_exact(sock, 4)
    (length,) = struct.unpack("!I", header)
    return recv_exact(sock, length)

This is not a crypto step.

It is a protocol step.

And that distinction matters more than it first appears. A secure channel is not just “call encrypt on a string.” It is a protocol with structure, state, and rules.

Now that we have a way to send and receive well-defined records, we can finally wrap them in encryption.

Step 3 — Wrapping the channel in shared-key encryption

The most obvious first attempt at secure communication is usually this:

• both sides already know the same secret key

• the sender encrypts the message before sending

• the receiver decrypts it after receiving

That is exactly what we will do.

No handshake yet.

No certificates yet.

No integrity yet.

No authentication yet.

This version is intentionally naive.

For encryption, I will use AES in CTR mode.

Very briefly:

• AES is a symmetric block cipher

• CTR mode makes it convenient for encrypting a stream of bytes

• it gives us confidentiality

• but it does not give us integrity

That last point is the important one for this article.

If you want a deeper explanation of AES itself, I already wrote about it in my cryptography series: https://www.dmytrohuz.com/p/building-own-block-cipher-part-3, so I will not go into the internals here.

The nonce

CTR mode also needs a nonce.

For now, think of it as a fresh per-message value that must be different for each encryption under the same key.

It is not secret.

It just must not be reused.

So our encrypted payload will look like this:

• nonce

• ciphertext

Crypto helper

# crypto.py
import os
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes

# 32-byte shared key for AES-256
SHARED_KEY = b"0123456789ABCDEF0123456789ABCDEF"

def encrypt_message(plaintext: bytes) -> bytes:
    nonce = os.urandom(16)

    cipher = Cipher(algorithms.AES(SHARED_KEY), modes.CTR(nonce))
    encryptor = cipher.encryptor()
    ciphertext = encryptor.update(plaintext) + encryptor.finalize()

    # Encrypted payload format:
    # nonce || ciphertext
    return nonce + ciphertext

def decrypt_message(payload: bytes) -> bytes:
    nonce = payload[:16]
    ciphertext = payload[16:]

    cipher = Cipher(algorithms.AES(SHARED_KEY), modes.CTR(nonce))
    decryptor = cipher.decryptor()
    plaintext = decryptor.update(ciphertext) + decryptor.finalize()

    return plaintext

At this point, our wire format becomes:

• 4-byte length

• 16-byte nonce

• ciphertext

+----------------+----------------+---------------------+
| length (4 B)   | nonce (16 B)   | ciphertext (N bytes)|
+----------------+----------------+---------------------+

That already looks much more like a protocol.

Step 4 — Encrypt the request and response

Now let’s integrate this into the client and server.

Encrypted client

# client_v1.py
import socket

from framing import send_record, recv_record
from crypto import encrypt_message, decrypt_message

HOST = "127.0.0.1"
PORT = 8081

request = (
    "GET /transfer?to=bob&amount=100 HTTP/1.1\\r\\n"
    "Host: localhost\\r\\n"
    "\\r\\n"
).encode("utf-8")

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client:
    client.connect((HOST, PORT))

    encrypted_request = encrypt_message(request)
    send_record(client, encrypted_request)

    encrypted_response = recv_record(client)
    response = decrypt_message(encrypted_response)

    print("Received decrypted response:")
    print(response.decode("utf-8"))

Encrypted server

# server_v1.py
import socket

from framing import send_record, recv_record
from crypto import encrypt_message, decrypt_message

HOST = "127.0.0.1"
PORT = 8081

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as server:
    server.bind((HOST, PORT))
    server.listen(1)

    print(f"Listening on {HOST}:{PORT}")
    conn, addr = server.accept()

    with conn:
        print(f"Connected by {addr}")

        encrypted_request = recv_record(conn)
        request = decrypt_message(encrypted_request)

        print("Received decrypted request:")
        print(request.decode("utf-8"))

        response = (
            "HTTP/1.1 200 OK\\r\\nContent-Type: text/plain\\r\\n"
            "Content-Length: 13\\r\\n\\r\\nhello, client"
        ).encode("utf-8")

        encrypted_response = encrypt_message(response)
        send_record(conn, encrypted_response)

Now the communication flow changes in an important way.

Instead of sending readable HTTP-like text directly, the client sends an encrypted record. The server reads the record, decrypts it, and sees the original request.

What changes on the wire

If you capture this version in Wireshark, the traffic is no longer readable.

Instead of clear request and response text, you now see opaque binary data.

So yes, we gained something real.

Let’s stop and say exactly what that is.

What encryption actually gave us

This first version gives us one meaningful property:

confidentiality against passive observers

If someone can only observe the traffic, but cannot modify it, they no longer get the plaintext request and response for free.

That is already better than raw TCP.

And this is why “just add encryption” feels so convincing. It visibly solves a real problem.

But that visible success can hide another, more dangerous failure.

Because a secure channel needs more than secrecy.

It also needs protection against tampering.

And we still do not have that.

Step 5 — Why encryption alone is not enough

This is the real point of Part 1.

We encrypted the messages.

We did not make them trustworthy.

AES-CTR protects confidentiality, but it does not protect integrity.

That means an active attacker may be able to modify ciphertext, and those modifications will flow through into the decrypted plaintext.

Very roughly, CTR mode behaves like this:

ciphertext = plaintext XOR keystream

So if an attacker changes bits in the ciphertext, the corresponding bits change in the plaintext after decryption.

That property is called malleability.

And protocol messages are usually predictable enough that this becomes useful to an attacker.

Our example request already has a very predictable structure:

GET /transfer?to=bob&amount=100 HTTP/1.1
Host: localhost

The exact bytes of amount=100 are not random.

That predictability is enough to hurt us.

A tiny isolated demo

We do not need a full man-in-the-middle proxy to show the problem. A small isolated example is enough.

# ctr_malleability_demo.py
from crypto import encrypt_message, decrypt_message

original = b"amount=100"
encrypted = encrypt_message(original)

nonce = encrypted[:16]
ciphertext = bytearray(encrypted[16:])

# Change '1' -> '9'
# ASCII '1' = 0x31
# ASCII '9' = 0x39
# Difference = 0x08

index_of_digit = len("amount=")
ciphertext[index_of_digit] ^= 0x08

modified = nonce + bytes(ciphertext)
decrypted = decrypt_message(modified)

print("Original :", original)
print("Modified :", decrypted)

Output:

Original : b'amount=100'
Modified : b'amount=900'

And that is the failure.

The attacker did not need the key.

They did not need to fully decrypt the message first.

They only needed the ability to modify the encrypted bytes in transit.

The receiver then decrypts the modified ciphertext and gets modified plaintext — without any built-in indication that anything went wrong.

So even though the message is hidden from passive observers, it is still vulnerable to active tampering.

That is not a secure protocol.

That is only encrypted transport.

What is still broken

At this point, our fake secure channel still has many serious holes.

No integrity protection

The receiver cannot detect that the ciphertext was modified.

No message authentication

The receiver has no cryptographic proof that the message came from the expected sender and arrived unchanged.

No replay protection

An attacker can capture an encrypted message and replay it later.

Static shared key

Both sides use one long-term shared key for everything.

That does not scale, and if it leaks, everything built on top of it collapses.

No handshake

There is no fresh session establishment. The peers do not negotiate anything. They just start encrypting.

No peer identity

The client does not really know who it is talking to beyond “someone who can decrypt with this key.”

So yes, we improved something.

But we are still very far from TLS.

Good.

That is exactly what Part 1 should make visible.

Summary

In this first part, we built a fake secure channel.

We started with plain socket communication and saw that everything was fully transparent. Then we wrapped the communication in shared-key encryption with AES-CTR, which gave us confidentiality against passive observers.

That was real progress.

But it was not enough.

Because encrypting a message is not the same thing as protecting the message from being changed. Our channel still accepts modified ciphertext, decrypts it, and trusts the result.

So the first lesson of this series is simple:

confidentiality is not integrity

And if we want something that starts to deserve the name secure protocol, we need both.

Next part — adding integrity with a MAC

So we stop here.

We now have a channel that can hide bytes from passive observers, but cannot reliably detect tampering.

In the next part, we will keep the same basic setup and fix the biggest hole we exposed here: we will add a MAC — a Message Authentication Code.

That will take us from:

“you probably can’t read this”

to:

“you also can’t silently change this”

That still will not be real TLS.

But it will make our fake secure channel one step less fake.

Final code

I’ll put the full code for this part on GitHub here: rebuilding_tls

Next article:

A year ago I wrote a series of articles about how a web server works.

I started from a very primitive version and step by step moved toward the same core ideas modern production servers rely on. That journey was already deep enough on its own. But when I finished it, I had one more thing in mind.

A small improvement.

Wrap the server in TLS. Make the communication secure.

At least, that was how it looked from the outside.

In reality, that “small improvement” turned into a much longer journey than I expected. What looked like one tiny feature hidden behind the familiar lock icon in the browser opened the door into a huge world of cryptography, key exchange, authentication, certificates, trust, protocol design, and many layers of details that usually stay invisible when everything just works.

And that is exactly why I decided to make this series.

Subscribe now

I did not want to approach TLS as a finished black box. I did not want to start from the RFC and just repeat the names of the protocol messages until they sounded familiar. I wanted to understand what TLS is, why it exists in the form it exists, and why it needs so many moving parts.

So this series is my attempt to rebuild it.

Not the full production-ready TLS implementation, of course. But a step-by-step reconstruction of the logic behind it. We will start from something intentionally naive, something that only looks secure, and in each article we will add one missing piece, one new idea, one more reason why real TLS had to become what it is.

By the end, I want us to arrive at a simplified TLS-like protocol that is close enough to the real thing to make the real thing much easier to understand.

Because that is still the real goal.

I do not want to stay forever in toy examples, and I do not think you want that either. In the final part of the series, I want to take everything we built ourselves and map it to the real TLS protocol: what is different, what extra problems the real one solves, why it is structured the way it is, and maybe also how some popular libraries implement it in practice.

But I really believe that jumping directly into the finished TLS protocol is the wrong way to learn it.

TLS is one of those technologies where the final design hides the reasons. If you look at the real protocol too early, you see a forest of details: handshakes, certificates, transcript hashes, traffic secrets, record protection, extensions, verification steps. All of them are there for a reason. But those reasons are much easier to understand when you first build the broken versions that fail without them.

That is the main idea of this series:

we will learn TLS by first building the wrong thing, and then fixing it step by step.

So if your goal is to understand the real TLS better, I would strongly recommend following the whole journey and not jumping directly to the last part.

As I mentioned, TLS is built on many cryptographic ideas. I will explain the important ones when we need them, but I also already wrote a separate series where I rebuild the main cryptographic foundations from scratch:

So throughout this TLS series, I will link back to those parts whenever we meet a concept that deserves a deeper look.

This page will be the central hub for the whole project. I will keep it updated as new parts are published, so all articles stay connected in one place.

I really hope you enjoy this journey as much as I do.

Full Summary of the TLS Series

This section will be updated as new parts are released.

Subscribe now

What this series is really about

This is not just a series about TLS.

It is also another exercise in the same thing I keep coming back to again and again: taking foundational technology that usually appears to us as a finished black box, opening it up, and rebuilding it from simpler parts until it stops feeling magical.

That was the idea behind the web server series.

That was the idea behind the cryptography series.

And now this is the same idea applied to TLS.

Follow the journey

This page will stay the central entry point for the whole series.

If this kind of deep, step-by-step reconstruction of systems is interesting to you, you can subscribe so you do not miss the next parts.

Links

• Part 4 walkthrough (rendered) — dmytrohuzz.github.io/rebuilding_tls/.../walkthrough.html

• Repository — github.com/DmytroHuzz/rebuilding_tls

• Questions, feedback, found a bug? — LinkedIn: dmitriyhuz

Subscribe now

Time looks simple until you have to trust it.

We use timestamps everywhere: logs, APIs, databases, schedulers, certificates, metrics, traces, distributed systems, industrial systems. But the moment you start asking basic questions — what exactly is this timestamp measuring? which clock produced it? how does one machine keep time? how do many machines agree on it? — the topic becomes much deeper than it first appears.

This series was my attempt to build a practical mental model of time for developers from first principles all the way down to Linux clocks, NTP, PTP, PHCs, and synchronization tools.

If you want one entry point into the whole topic, this is it.

Share

What this series covers

This series is built as a path:

• first, what time actually means in software

• then, how one computer keeps time

• then, how many computers synchronize time

• and finally, how Linux represents all of this in practice

The goal was never to produce a dry reference manual.

The goal was to make time feel understandable.

Not just “I know NTP exists,” but a real working model of:

• what time is

• how clocks drift

• why synchronization is hard

• why timestamp location matters

• and how Linux exposes all of this in real systems

The full reading path

Subscribe now

This is the foundation.

If you ever used timestamps without being fully sure what they really mean, start here.

This part covers the basic language of the topic: what time means in software, why timestamps are not “time itself,” what role standards and agreements play, and why the whole subject is more subtle than it first appears.

This is the part that gives you the mental vocabulary for everything that comes later.

Once the theory is clear, the next question is obvious:

How does one computer actually keep time?

This part moves inside the machine. It covers the clocks and mechanisms Linux uses to track time, including the RTC, system time, counters, and the distinction between different clock sources and time domains.

If Part 1 explains what time means, Part 2 explains how a single system turns that idea into something measurable and usable.

A single computer can keep time locally.

A distributed system has a harder problem: many machines must keep time together.

This part is about synchronization. Why simply setting clocks once does not work. Why drift makes synchronization a continuous process. How NTP and PTP approach the problem. And why the quality of synchronization depends not only on the protocol, but also on where timestamps are taken.

This is where time stops being a local machine detail and becomes a systems problem.

This is the practical payoff.

It turns the concepts from the whole series into the Linux view of the world:

• RTC

• system clock

• PHC

• NTP

• PTP

• ptp4l

• phc2sys

• and the usual synchronization paths between them

This is the compact field guide version — the part you can actually bookmark and return to when you need to inspect, operate, or debug time synchronization on Linux.

Interactive visuals and supporting materials

Along the way, I also started building visual and interactive explanations for some of the harder ideas, such as:

• clock drift

• synchronization by timestamp exchange

• NTP principles

• PTP principles

• software vs hardware timestamping

I want this series to be more than just text. Time is easier to understand when you can see it move.

Who this series is for

This series should be useful if you work with:

• backend or distributed systems

• Linux and infrastructure

• observability and logs

• event ordering

• industrial or measurement systems

• networking

• NTP/PTP

• or just any system where timestamps need to be trusted

In other words: if time can break your system, this topic is worth understanding properly.

Why I wrote this

Time is one of the most used and least understood parts of software.

Most of us interact with it every day, but only occasionally stop to ask what is actually happening underneath. I wanted to fix that for myself first — and then turn that learning process into something practical and usable for other engineers.

This series is the result.

Final note

If you made it through the whole series, you did not just read a few posts about clocks.

You built a real mental model of time in computing — from first principles, to clocks inside a machine, to synchronization across networks, to the actual Linux entities and tools that make it work in practice.

That already puts you ahead of most engineers who touch these systems.

You now know enough to stop treating time as a mysterious background feature and start seeing it for what it really is:

infrastructure, measurement, coordination, and engineering.

Bookmark this page. I’ll keep it as the main entry point for the whole series.

If you got here and made it through the previous articles, you have already done the hard part. You are basically a time guru now.

You know what time means in computing, how a machine keeps it, why clocks drift, how synchronization works, and why timestamp location matters. That is already more than most people ever learn about this topic.

Now let’s compress all of that into the Linux view of the world.

This is the top of the iceberg: a small set of clocks, commands, and tools that represent most of the concepts we have been building up across the series.

Think of this as the practical cheat sheet — the 90% version. The one you can use to inspect clocks, understand what is synchronized to what, and handle most everyday Linux time-sync tasks without drowning in documentation.

This is probably the part you will want to bookmark.

Share

The three main clock entities in Linux

When people say “Linux time,” they often mean one thing. In reality, Linux commonly deals with at least three different clock entities:

• system time

• RTC

• PHC

They serve different purposes.

1. System time

This is the normal wall-clock time used by most applications.

It is what you usually see when you run:

date

or:

timedatectl

Conceptually, this is the kernel’s main wall clock, commonly associated with CLOCK_REALTIME.

This is the clock used by:

• most user-space applications

• logs

• system services

• everyday time queries

Quick check

date
timedatectl

Mental model

System clock = the main OS wall clock

2. RTC (Real-Time Clock)

The RTC is the battery-backed hardware clock on the motherboard.

Its main job is simple: keep time while the machine is powered off.

Linux often uses it during boot to initialize system time, and may update it again later from system time. But the RTC is usually not the main precision synchronization clock during normal operation.

Quick check

sudo hwclock --show
timedatectl

Common operations

Copy system time to RTC:

sudo hwclock --systohc

Copy RTC to system time:

sudo hwclock --hctosys

Mental model

RTC = persistent clock for boot/shutdown

3. PHC (PTP Hardware Clock)

A PHC is a hardware clock exposed by a PTP-capable network interface.

This is where Linux gets especially interesting.

A PHC lives on the NIC, much closer to the real transmit/receive event than the normal system clock. That is why it matters for precise synchronization.

PHCs usually appear as device files like:

/dev/ptp0
/dev/ptp1

Quick check

List PHC devices:

ls -l /dev/ptp*

Check which PHC belongs to a NIC and whether hardware timestamping is supported:

ethtool -T eth0

Mental model

PHC = hardware clock on the NIC, used for precise packet timing

One picture: how they relate

RTC
  |
  | used mainly at boot / shutdown
  v
System clock (CLOCK_REALTIME)
  ^
  |
  | phc2sys can sync between them
  |
PHC (/dev/ptpX on NIC)
  ^
  |
  | ptp4l syncs PHC to PTP network
  |
PTP network / Grandmaster

A rough summary:

• RTC keeps time while power is off

• system clock is what most software reads

• PHC is the precision clock on the NIC

How Linux syncs time with NTP

In the NTP world, Linux usually synchronizes the system clock.

Common tools:

• chronyd

• systemd-timesyncd

• older setups may use ntpd

Check whether NTP is active

timedatectl

If you use chrony

Show synchronization state:

chronyc tracking

Show time sources:

chronyc sources -v

Mental model

NTP servers
   |
   v
chronyd / timesyncd / ntpd
   |
   v
System clock

In other words, NTP usually targets the system clock, not the PHC.

How Linux syncs time with PTP

In the PTP world, Linux often follows a two-step path:

1. synchronize the PHC to the PTP network

2. synchronize the system clock to that PHC

The two main tools are:

• ptp4l

• phc2sys

ptp4l: syncing the NIC-side clock

ptp4l speaks PTP on the network and usually synchronizes the PHC associated with the interface.

Typical example:

sudo ptp4l -i eth0 -m

Meaning:

• i eth0 → use interface eth0

• m → print log messages to stdout

Mental model

PTP Grandmaster / network
        |
        v
      ptp4l
        |
        v
PHC on eth0 (/dev/ptpX)

This is usually where the NIC-side precision timing gets aligned to the network timing domain.

phc2sys: syncing one local clock to another

Once the PHC is synchronized, the rest of the machine may still be reading the system clock.

That is why phc2sys exists.

Its job is to synchronize one local clock to another.

The most common use is:

PHC → system clock

Example:

sudo phc2sys -s eth0 -c CLOCK_REALTIME -m

Meaning:

• s eth0 → source is the PHC associated with eth0

• c CLOCK_REALTIME → target is the system clock

• m → print status

Mental model

PHC on NIC
   |
   v
phc2sys
   |
   v
System clock

This is the classic Linux PTP flow.

The classic Linux PTP pipeline

PTP Grandmaster
      |
      v
  [ network ]
      |
      v
NIC hardware timestamping
      |
      v
    ptp4l
      |
      v
PHC (/dev/ptp0) synchronized to PTP domain
      |
    phc2sys
      |
      v
System clock (CLOCK_REALTIME)
      |
      v
Applications / logs / services

This is one of the most useful diagrams to keep in your head.

PHC to system clock, or system clock to PHC?

In precision setups, the common direction is:

PHC → system clock

because the PHC is closer to the wire and usually the better timing source in a PTP environment.

That is why this is a common command:

sudo phc2sys -s eth0 -c CLOCK_REALTIME -m

But phc2sys is more general than that. It can synchronize clocks in other directions too.

ts2phc:Syncing PHCs

Linux can synchronize hardware clocks too, but the right tool depends on the synchronization path.

If the goal is to make the system clock follow a PHC, the usual tool is phc2sys. If the goal is to synchronize one or more PHCs from an external timestamp source such as PPS or GNSS, the usual tool is ts2phc. ts2phc is specifically designed to synchronize PHCs to external timestamp signals, and it can distribute one source to multiple PHCs.

Conceptually:

External PPS / GNSS / timestamp source → ts2phc → one or more PHCs
PHC → phc2sys → system clock

A typical ts2phc command looks like this:

sudo ts2phc -s eth0 -c eth1 -m

In this form:

• -s eth0 selects the source clock or source interface,
  

• -c eth1 selects a PHC to synchronize,
  

• -m prints log messages to stdout. The tool also allows multiple -c options if you want to synchronize more than one PHC from the same source.
  

This is less common than PHC-to-system-clock synchronization, but it matters in systems where multiple hardware clocks need to follow the same precise external reference.

Where software timestamping fits

Not every NIC has hardware timestamping. Not every system needs that level of precision.

Linux can still synchronize clocks using software timestamps, and for many use cases that is completely fine.

But the practical tradeoff remains the same:

• hardware timestamping gives measurements closer to the real wire event

• software timestamping includes more delay and variation from the OS path

That is why software timestamping usually belongs to a looser precision budget, while hardware timestamping is what unlocks much tighter synchronization.

The most useful commands at a glance

Check system time

date
timedatectl

Check RTC

sudo hwclock --show

List PHC devices

ls -l /dev/ptp*

Check NIC timestamping support

ethtool -T eth0

Run PTP on an interface

sudo ptp4l -i eth0 -m

Sync system clock from PHC

sudo phc2sys -s eth0 -c CLOCK_REALTIME -m

Sync PHCs

sudo ts2phc -s eth0 -c eth1 -m

Check chrony state

chronyc tracking
chronyc sources -v

The one table worth remembering

RTC ------------------> system time at boot / shutdown
NTP daemon -----------> system clock
ptp4l ----------------> PHC
phc2sys --------------> PHC <-> system clock
External PPS / GNSS / timestamp source → ts2phc → one or more PHCs
hwclock --systohc ----> system clock -> RTC
hwclock --hctosys ----> RTC -> system clock

The biggest practical lesson

When somebody says:

“The machine is synchronized.”

That is usually too vague.

The useful follow-up question is:

Which clock is synchronized?

• the RTC?

• the system clock?

• the PHC?

• and which one is the application actually using?

That one question prevents a lot of confusion.

One-screen summary

RTC
- battery-backed motherboard clock
- keeps time while machine is off
- checked with: hwclock --show

System clock
- main OS wall clock
- used by most applications
- checked with: date, timedatectl
- synchronized by: NTP or by phc2sys from PHC

PHC
- hardware clock on a PTP-capable NIC
- represented as: /dev/ptpX
- checked with: ls /dev/ptp*, ethtool -T eth0
- synchronized by: ptp4l

Typical Linux PTP flow
PTP network -> ptp4l -> PHC -> phc2sys -> system clock

Final note

If you made it all the way here, you did not just read a few articles about clocks.

You built a real mental model of time in computing — from first principles, to clocks inside a machine, to synchronization across networks, to the actual Linux entities and tools that make it work in practice.

That already puts you far ahead of most engineers who touch these systems.

You now know enough to stop treating time as a mysterious background feature and start seeing it for what it really is: infrastructure, measurement, coordination, and engineering.

And it was the final piece: a practical cheat sheet you can actually use. Bookmark it. Return to it. Break things with it. Fix things with it.

Intro

Every action film has that scene just before the military operation begins.

“Let’s sync our watches,” the captain says.

The idea is simple: if every stage of the plan depends on precise coordination, everyone involved has to act in sync and according to the same timeline.

A while ago, we started our journey with a practical goal: synchronizing the time of many computers. To get there, we first had to understand what time actually is and learn the basic glossary needed to speak the language of this problem and its solutions. In the first part (https://www.dmytrohuz.com/p/a-practical-guide-to-time-for-developers), we explored the foundations of time itself. In the second (https://www.dmytrohuz.com/p/a-practical-guide-to-time-for-developers-2ec), we looked at how time is kept and tracked inside a single computer. Now we are finally ready to move to the next step: how many computers share time with each other.

Keeping precise time across many computers is not unusual or exotic. In fact, the opposite is true. Distributed systems, industrial networks, telecom infrastructure, financial systems, and measurement environments often involve hundreds or thousands of devices that must stay synchronized within a clearly defined precision budget.

Let’s imagine a wind farm. Each turbine is around 120 meters tall and has a warning light at the top. To make the turbines visible to planes at night, the lights should blink every second. And to make the whole field clearly visible as one coordinated structure, those lights should blink simultaneously.

How can we make that happen?

The obvious answer is: the turbines need synchronized clocks.

But how we can keep them in sync for hundreds and thousands devices with amazing accuracy?

Let’s see!

Just sync the clocks once?

Let’s start with the most obvious idea: set the same time on all clocks once, and the problem is solved.

Unfortunately, it does not work that way.

Every clock has physical behavior behind it. Its frequency is affected by things like oscillator quality, temperature, aging, and other environmental factors. As a result, every clock drifts in its own way. Some also exhibit short-term fluctuations, often described as wander. These effects cannot be fully eliminated, and in practice they mean that two clocks will slowly diverge even if they start perfectly aligned.

That turns synchronization from a one-time setup task into a continuous process.

Clocks do not just need to be set. They need to be kept aligned over time. In practice, that means measuring the difference between clocks again and again, then adjusting their time and, more importantly, their rate so that they do not immediately drift apart again.

You can see how quickly clocks with different rates and wander fall out of sync, even when they start at exactly the same time:

Feel free to explore the interactive simulation I created for this exact scenario and see how quickly clocks drift out of sync: https://dmytrohuzz.github.io/interactive_demo/clock_sync/index.html

From setting time to synchronization

Once we accept that clocks drift, a one-time setup stops looking like a real solution. Time is not something you assign once. It is something you keep aligned.

In practice, synchronization is a feedback loop. A machine compares its local clock to some reference, estimates the difference, adjusts its own clock, and repeats the process again and again.

The difficult part is that machines cannot read each other’s clocks directly. They can only communicate over a network, and the network adds delay and uncertainty. So synchronization protocols work indirectly: they exchange messages with timestamps and use those timestamps to estimate the relationship between clocks.

At the center of that estimate are two questions:

• how far apart are the clocks?

• how much of the observed difference comes from network delay rather than clock error?

This sounds simple in theory, but the key idea only becomes clear once we walk through it step by step.

A basic synchronization exchange gives us four timestamps:

• t1 — the client sends a request

• t2 — the server receives that request

• t3 — the server sends a response

• t4 — the client receives the response

These four timestamps are the heart of the whole mechanism. Once this pattern becomes intuitive, the rest of the synchronization topic becomes much easier to follow.

Now imagine the exchange from the client’s point of view.

The client sends a request at local time t1 = 00:00.

Later, it receives the response at local time t4 = 00:04.

Inside that response, the server includes its own timestamps:

• it received the request at t2 = 00:06

• it sent the response at t3 = 00:06

At first glance, this looks strange. How can the server receive the request at 00:06 if the client sent it at 00:00, and the whole round trip took only four seconds on the client side?

The answer is simple: t1 and t2 do not belong to the same timeline.

The client clock and the server clock are different local views of time. What synchronization tries to estimate is the relation between those two timelines. In other words, it tries to answer this question:

If the client sees one moment as 00:00, what does the server call that same moment?

That relationship is what we call offset.

This is the most important insight in the whole topic: the difference t2 - t1 does not represent only network delay. It contains two things mixed together:

• packet travel time

• clock offset between client and server

A useful way to think about it is with time zones.

Imagine you leave one city at local time 00:00, travel to another city, and arrive when the local clock there shows 14:00. Then you immediately turn around and come back, arriving home when your original city’s clock shows 20:00.

Now suppose the travel time is the same in both directions.

The first leg, from your city to the other one, includes:

travel time + time-zone difference

The return leg includes:

travel time - time-zone difference

So if the outward journey appears shorter or longer than the return journey, that difference tells you something about the offset between the two local clocks.

This is exactly what synchronization protocols exploit.

Under the usual symmetric-delay assumption, the offset can be estimated as:

offset = ((t2 - t1) + (t3 - t4)) /2

and the round-trip delay as:

delay = (t4 - t1) - (t3 - t2)

The first formula separates clock offset from the two directions of travel. The second removes the server’s processing time and leaves only the network round-trip time.

So synchronization is not about directly copying time from one machine to another. It is about observing message exchanges, separating delay from clock difference, and then correcting the local clock based on that estimate.

That is the core idea behind the whole topic.

Feel free to play with the simulation here:

https://dmytrohuzz.github.io/interactive_demo/clock_sync/clock_sync_explained

NTP and PTP: two ways to synchronize clocks

Over time, two major protocol families became the standard answers to the synchronization problem: NTP and PTP.

Both solve the same core problem: a machine cannot read another machine’s clock directly, so it has to infer the difference by exchanging timestamped messages over a network. From those timestamps, it estimates clock offset and network delay, then adjusts the local clock toward a reference.

The difference is not the basic idea, but the precision target and the environment they are designed for.

NTP: practical synchronization for general systems

NTP — the Network Time Protocol — is the general-purpose approach. It is designed to keep clocks reasonably aligned across ordinary systems and ordinary networks.

Its main principle is simple: a client exchanges request and response messages with a time server, records timestamps on both sides, estimates round-trip delay and clock offset, and then gradually disciplines its own clock. It repeats this process continuously, using multiple measurements to smooth out noise and avoid reacting too aggressively to one bad sample.

That makes NTP a good fit for:

• logs and observability

• authentication and certificate validation

• scheduled jobs

• general wall-clock correctness across servers and infrastructure

NTP does not assume a perfect network. It is built for real environments, where delays vary, paths are not perfectly symmetric, and hosts are under changing load. Its strength is robustness, not extreme precision.

[Interactive Demo]

PTP: tighter synchronization for controlled environments

PTP — the Precision Time Protocol — targets systems where much tighter agreement between clocks is required.

Its principle is similar to NTP: devices exchange timing messages, estimate offset and delay, and adjust local clocks. But PTP is designed for local precision networks, where the entire timing path is treated more carefully. In practice, this often means hardware timestamping, PTP-aware switches, and a dedicated timing hierarchy built around a grandmaster clock distributing time to other devices.

PTP is commonly used in:

• industrial and automation systems

• telecom networks

• audio and video systems

• measurement systems

• finance

• power and substation environments

PTP is not just “a more accurate NTP.” It usually operates in a different class of environment, with tighter timing requirements and more deliberate infrastructure support.

[Interactive Demo]

Different tools for different timing budgets

So NTP and PTP are not really rivals. They are different engineering choices.

If the goal is to keep ordinary systems aligned to real time well enough for general infrastructure behavior, NTP is usually the right tool.

If the goal is to keep clocks tightly aligned in a local timing domain where timing quality directly affects correctness, event ordering, or measurement precision, PTP is often the better fit.

The key point is this: both protocols depend on timestamp exchange, but the quality of synchronization depends heavily on how those timestamps are produced.

And that leads to the next question: where exactly was the timestamp taken?

This is where timestamping location — in software or in hardware — starts to matter.

Why timestamp location changes everything

At this point, NTP and PTP may still look like protocol problems: exchange messages, estimate offset, correct the clock.

But in practice, a large part of synchronization quality depends on something more physical:

where exactly is the timestamp taken?

That matters because a packet does not appear in software at the exact moment it hits the wire. Between the real network event and the moment the operating system records a timestamp, the packet may pass through the NIC, driver, kernel, interrupt handling, scheduling, and software processing. Every one of those layers can add delay and variation.

So two timestamps may look equally precise as numbers while representing very different physical moments.

Software timestamping

With software timestamping, the timestamp is recorded somewhere in the software stack after the packet has already passed through part of the system.

That makes software timestamping widely available and easy to use, but it also means the measurement includes more uncertainty:

• interrupt latency

• kernel and driver delay

• scheduling effects

• queueing and system load

As a result, a software timestamp often reflects when the system handled the packet, not the exact moment the packet crossed the network interface.

Hardware timestamping

With hardware timestamping, the timestamp is recorded much closer to the real transmit or receive event, typically inside the NIC itself.

This removes a large part of the software-induced uncertainty and makes the measurement more stable and repeatable. The closer the timestamp is to the actual wire event, the more useful it becomes for precise synchronization.

That is one of the main reasons PTP can achieve much better accuracy in the right environment: not only because of the protocol itself, but because it is often paired with hardware timestamping and a more carefully controlled timing path.

So the practical precision limit is not defined by the protocol name alone. It depends on the full measurement path.

A good rule of thumb is simple:

the closer the timestamp is to the wire, the better the synchronization can be.

Summary

A single computer can keep time locally. A distributed system has a harder task: many machines must keep time together.

That is why simply setting clocks once is not enough. Real clocks drift, so synchronization has to be continuous. Protocols such as NTP and PTP address this by exchanging timestamped messages, estimating clock offset and network delay, and repeatedly steering local clocks toward a reference.

But protocol choice is only part of the story. In practice, synchronization quality also depends heavily on where timestamps are taken. A timestamp captured deep in software carries more uncertainty than one captured close to the physical network event.

So if this part was about the general idea of shared time — why it matters, why it is difficult, and how systems approach it — the next part will move from principle to implementation.

We will look at how Linux actually does this in practice: NICs, software and hardware timestamping, PHCs, and the tools that connect them into a real synchronization stack.

Next part:

We like to talk about “the cloud” as if software has escaped geography.

The interface encourages that illusion. You choose a region, deploy a service, replicate some data, add a failover plan, and the system starts to feel abstract. Compute is elastic. Storage is managed. Infrastructure appears to have become location-independent.

But critical systems do not really stop living somewhere.

They still sit on top of power, cooling, fiber, telecom topology, legal jurisdiction, operational teams, and the physics of latency. And the more a system becomes real — real users, real regulation, real response-time constraints, real business consequence — the more geography tends to re-enter the design.

That is the part cloud culture hides well: cloud abstracts geography at the interface level, but many important systems reintroduce geography at the architecture level.

Recent disruptions around AWS infrastructure in the Gulf make that harder to ignore. Reuters reported issues affecting AWS data centers in the UAE and Bahrain amid Iranian strikes, including problems tied to power and connectivity. Separate Reuters reporting also described incidents in the UAE involving drone interceptions and falling debris in Fujairah. Whether one looks at these as isolated disruptions or as signals of a broader shift, the underlying point is the same: data centers are no longer just “IT facilities.” They increasingly sit inside the same map of strategic exposure as energy, ports, and communications infrastructure.

That does not just make cloud infrastructure more important. It makes geography more important.

The abstraction is useful. The abstraction is also misleading.

This is not an anti-cloud argument.

Cloud abstractions are powerful because they compress operational complexity. They let teams build and scale systems without owning every layer directly. They make certain kinds of resilience easier to implement. They lower coordination cost. They make global software feel tractable.

But useful abstractions have a habit of concealing the substrate.

In cloud, the concealed substrate is not just hardware. It is geography.

A lot of modern software is designed as if “where it runs” is a secondary implementation detail. For some workloads, that is mostly true. For critical ones, it often is not.

Latency-sensitive systems want to be physically closer to where requests originate. Regulated systems want data to remain in specific jurisdictions. Operationally important systems want predictable local integration with identity, networking, observability, and response teams. Once those requirements become strong enough, the architecture starts to pull back toward place.

So while the cloud sells a kind of placelessness, critical systems often buy proximity instead.

Latency quietly defeats the fantasy of placeless compute

The first force that brings geography back is latency.

Latency is often described as just a technical metric. In practice, it is a design constraint that pushes infrastructure back into physical space. If response time matters, distance matters. If user experience matters, path length matters. If control loops matter, locality matters.

This is not theoretical. AWS explicitly recommends choosing regions close to users for latency reasons, and sells Local Zones and Wavelength for workloads that need to run nearer to end users and telecom networks. Microsoft says similar things about Azure Extended Zones for low-latency and data-residency-sensitive workloads.

At the interface level, cloud encourages us to think in regions, services, and APIs. At the architecture level, latency-sensitive systems often say something much simpler:

put the system near the place where the consequences happen.

That is already a partial collapse of abstraction.

Sovereignty brings geography back a second time

The second force is law.

Even if a workload could technically run anywhere, that does not mean it is allowed to. Data residency, sector-specific regulation, national security requirements, and jurisdictional constraints all push architecture back toward geography. That is why sovereign cloud offerings now exist at all. AWS has launched a European Sovereign Cloud specifically for stricter residency and operational autonomy requirements, while Google and Microsoft document controls for customers that cannot treat geography as interchangeable.

This is a useful correction to one of cloud’s most seductive promises.

People say cloud gives you geographic flexibility. Sometimes it does. But in important systems, law can be just as constraining as physics. The workload may appear abstract, but its permitted runtime geography can still be narrow.

Once again, cloud did not remove geography. It pushed geography behind a cleaner control plane.

The result is hidden concentration

This is where the systems point matters.

Cloud encourages a mental model of distribution. Critical systems often rebuild concentration underneath that model.

Not recklessly. Often for completely rational reasons:

• lower latency

• data residency

• operational simplicity

• cost structure

• regional business requirements

• local ecosystem dependence

Each decision can make sense in isolation. But in aggregate, a system may look globally abstract while the important runtime, data, and dependency paths remain tied to a much narrower geography.

That is where hidden fragility comes from.

The problem is not that cloud is fake. The problem is that cloud can make concentration feel more diversified than it really is.

A clean interface can hide a concentrated substrate.

And once that substrate becomes strategically important, the illusion gets expensive.

That is why the AWS example in the Gulf matters beyond the specific incident. It is not only a story about one provider or one region. It is a visible reminder that cloud infrastructure now sits close enough to the center of commerce, communications, and increasingly AI-related compute demand that it begins to resemble critical infrastructure in the older sense of the term. And critical infrastructure is always geographic.

Good architecture can counter this — but not by accident

Cloud does not automatically imply weak geographic resilience. In fact, major providers explicitly support multi-region architectures, active-active designs, and cross-region failover. Google recommends multi-region deployments to improve latency and availability, and AWS Well-Architected explicitly warns against consolidating all workload resources into one geographic location. AWS also documents active-active multi-region patterns for low-latency and high-availability systems.

Cloud does not diversify geography by default just because it is cloud.

That has to be designed.

And the design often runs against real pressures:

• performance wants locality

• regulation wants jurisdictional specificity

• operations want manageable complexity

• economics want concentration where scale is cheapest

Resilience is not the natural resting state. It is a deliberate counterweight to optimization pressure.

That is the systems lesson.

Why this matters more now

This matters more than it used to because modern systems are becoming more dependent on concentrated compute.

As more of business, communications, platforms, and AI workloads sit on top of large cloud regions and data center clusters, the abstraction becomes more consequential — and more dangerous to misunderstand. The cloud did not make geography disappear. It made geography easier to ignore until latency, law, outage, or conflict forces it back into view.

At that point, the important question is no longer whether cloud is “really distributed.”

The better question is:

How much geographic reality has your architecture quietly reintroduced underneath the abstraction — and have you designed resilience there on purpose, or just assumed the word cloud already did that for you?

Because that assumption is where a lot of hidden concentration begins.