You Need the Time to Get the Time

For a while, every time I turned on my laptop, the clock said 27 July. It was the middle of October. The only way to fix it was to open Settings, switch Automatic Date & Time off, and switch it back on. Then the correct time appeared in a couple of seconds.

That fix is a strong clue. If restarting a service fixes the problem instantly, then the network, the servers and the configuration are all fine. Something goes wrong at boot and the service never recovers on its own. So I went through the logs to find out what.

Step 1: The Hardware Clock Is Lying

The first thing to check is the hardware clock (RTC), the small battery-backed clock on the motherboard that keeps time while the computer is off. The kernel prints its value at boot:

$ journalctl -b -k | grep "setting system clock"
rtc_cmos 00:00: setting system clock to 2023-03-19T00:01:13 UTC (1679184073)

2023. The boot before it read the correct date, so the RTC had reset itself while the laptop was off. That's the classic sign of a weak CMOS battery. I already knew mine was dying, so that part wasn't a surprise.

So where does "27 July" come from?

systemd won't accept a clock that is older than a built-in minimum date. If the RTC is earlier than that date, systemd moves the clock forward to it. On my system that minimum is 27 July 2026. So "27 July" doesn't mean anything by itself. It's just the oldest date systemd allows.

A dead CMOS battery explains the wrong time at boot. It doesn't explain why the automatic time sync never fixed it, which is the whole point of having it.

Step 2: How Ubuntu Syncs Time Now

On recent Ubuntu releases, the default time-sync service is chrony instead of systemd-timesyncd. It also uses NTS (Network Time Security) by default. Classic NTP is unauthenticated. NTS adds a TLS key exchange on TCP port 4460, so a man-in-the-middle can't feed you a fake time.

The default sources file looks like this:

# /etc/chrony/sources.d/ubuntu-ntp-pools.sources
pool 1.ntp.ubuntu.com iburst maxsources 1 nts prefer
pool 2.ntp.ubuntu.com iburst maxsources 1 nts prefer
pool 3.ntp.ubuntu.com iburst maxsources 1 nts prefer
pool 4.ntp.ubuntu.com iburst maxsources 1 nts prefer
pool ntp-bootstrap.ubuntu.com iburst maxsources 1 nts certset 1

NTS creates a chicken-and-egg problem. TLS needs a roughly correct clock to check certificates, and you're using NTS to get a correct clock. If your clock says July, every certificate issued after July looks "not yet valid" and the handshake fails.

Ubuntu planned for this. That's what the last line is for. ntp-bootstrap.ubuntu.com is checked against a special certificate set (certset 1, from /etc/chrony/nts-bootstrap-ubuntu.crt) that still works when the local clock is badly wrong. In theory, a machine that boots in the past should use the bootstrap server, step the clock, and then switch to the normal pools.

So why didn't that happen?

Step 3: Reading the chrony Logs

I first assumed the problem was chrony's makestep 1 3 setting, which only allows clock jumps during the first three updates. I was wrong. That's why it pays to count log events instead of skimming them.

This boot had four chronyd processes: one started at boot, and three more started when I toggled the setting. I counted the log messages for each one:

J=$(journalctl -b -u chrony -o short-monotonic)
for p in $(echo "$J" | grep -oP 'chronyd\[\K[0-9]+' | sort -un); do
  echo "== PID $p"
  echo "$J" | grep "chronyd\[$p\]" \
    | sed -E 's/.*chronyd\[[0-9]+\]: //; s/\[[0-9a-f:]+\]|[0-9]+(\.[0-9]+){3}/ADDR/g' \
    | sort | uniq -c | sort -rn | head
done

The instance started at boot (7 s → 423 s): never synced

     88 Could not connect to ADDR:4460 (ntp-bootstrap.ubuntu.com)
     45 Could not connect to ADDR:4460 (1.ntp.ubuntu.com)
     45 Could not connect to ADDR:4460 (2.ntp.ubuntu.com)
     ...
      4 TLS handshake with ADDR:4460 (N.ntp.ubuntu.com) failed : ...
        The certificate chain uses not yet valid certificate.

It never logged Selected source and never logged System clock wrong by. For seven minutes it didn't get a single usable time measurement.

The instance after my toggle (436 s): fixed within 5 seconds

TLS handshake ... (1..4.ntp.ubuntu.com) failed : ... not yet valid certificate.
Selected source 185.125.190.121 (ntp-bootstrap.ubuntu.com)
System clock wrong by 6917596.704108 seconds
System clock was stepped by 6917596.704108 seconds

This second instance stepped the clock by 6,917,596 seconds (about 80 days) right away, well within the makestep limit. So makestep wasn't the problem. The question was why the boot instance never got that far.

Step 4: Check Which Addresses Were Tried

Next I listed which addresses the boot instance actually tried to connect to:

BOOT_PID=$(journalctl -b -u chrony | grep -oP 'chronyd\[\K[0-9]+' | head -1)
journalctl -b -u chrony | grep "chronyd\[$BOOT_PID\]" \
  | grep -oE '(\[[0-9a-f:]+\]|[0-9]+(\.[0-9]+){3}):4460 \([^)]*\)' \
  | sort | uniq -c

     48 [2620:2d:4000:1::1123]:4460 (ntp-bootstrap.ubuntu.com)
     48 [2620:2d:4002:1::1123]:4460 (ntp-bootstrap.ubuntu.com)
     49 [2620:2d:4000:1::2123]:4460 (1.ntp.ubuntu.com)
     49 [2620:2d:4000:1::3123]:4460 (2.ntp.ubuntu.com)
     49 [2620:2d:4002:1::2123]:4460 (3.ntp.ubuntu.com)
     48 [2620:2d:4002:1::3123]:4460 (4.ntp.ubuntu.com)
      1 185.125.190.122:4460 (1.ntp.ubuntu.com)
      1 185.125.190.123:4460 (2.ntp.ubuntu.com)
      1 91.189.91.112:4460  (3.ntp.ubuntu.com)
      1 91.189.91.113:4460  (4.ntp.ubuntu.com)

That's the answer. The bootstrap server was only ever tried on IPv6 addresses. Is IPv6 even available here?

$ ip -6 route show default
$

There's no IPv6 default route. My network is IPv4-only, so every one of those 96 attempts was doomed. The four IPv4 connections it did make went to the normal pools, and those correctly failed with "not yet valid certificate", because the clock said July.

So the boot instance was stuck. The normal servers were reachable over IPv4, but their certificates couldn't be validated with the wrong date. The bootstrap server, the one source that works with a wrong date, was only tried over IPv6, which doesn't work on this network.

chronyd kept retrying the same IPv6 addresses and never looked up the bootstrap server again to get an IPv4 address. The NetworkManager hook (20-chrony-onoffline) did mark sources offline and online again at about 5 minutes after boot, but that didn't change which addresses chrony used.

When I toggled the setting, the new chronyd looked up the server names again with the network fully up. ntp-bootstrap.ubuntu.com resolved to an IPv4 address, and the clock was fixed within five seconds.

Why It Needs All Three Conditions

The bug only shows up when three things happen together. First, the clock is wrong at boot (a weak CMOS battery, a Raspberry Pi, some VMs), so every normal NTS server fails certificate checks. Second, the network is IPv4-only while the DNS records include IPv6 addresses. Third, the bootstrap pool uses maxsources 1, so chrony keeps one address at a time and never falls back to the other address family.

Take away any one of them and it works. With a correct clock, the normal pools succeed over IPv4. With working IPv6, the bootstrap connects. With more bootstrap sources, one of them is likely to be IPv4. That's probably why it isn't widely reported yet.

Workarounds

Option 1: make chrony IPv4-only. This is good if your network has no IPv6. Ubuntu's chrony unit reads its options from /etc/default/chrony:

# /etc/default/chrony
DAEMON_OPTS="-F 1 -4"

sudo systemctl restart chrony

Option 2: add an extra bootstrap source so chrony has more than one address to try. Put a drop-in file in /etc/chrony/sources.d/:

# /etc/chrony/sources.d/bootstrap-extra.sources
pool ntp-bootstrap.ubuntu.com iburst maxsources 2 nts certset 1

sudo chronyc reload sources

Option 3: replace the CMOS battery. This removes the trigger completely. It's a CR2032 coin cell (on some laptops a small wired cell) and costs a couple of dollars.

Reporting It Upstream

I'm reporting this against the chrony package on Launchpad, with the log evidence above. It could be fixed in more than one place. chrony itself could re-resolve a pool source when the NTS key exchange fails with "could not connect", or replace it with an address from the other address family. Ubuntu's default config could use maxsources 2 for the bootstrap pool, or add an IPv4-only bootstrap entry. And the NetworkManager hook could run chronyc refresh when connectivity appears, so the pools get resolved again.

If your Ubuntu machine boots with the wrong date and only fixes itself when you toggle automatic time, run the two commands from Step 4. If you see a long list of IPv6 addresses for ntp-bootstrap.ubuntu.com and no IPv6 default route, it's this bug.

Takeaways

The first lesson is that if a restart fixes something, the bug is in the service's starting state, not in its configuration, so compare the logs of the instance that failed with the one that worked. The second is to count log lines instead of skimming them. sort | uniq -c showed the IPv6-only pattern right away, while my first explanation (makestep) sounded right but wasn't supported by the numbers. The last is that secure protocols have startup problems of their own. NTS is a real improvement over plain NTP, but it depends on a roughly correct clock, and when the fallback for that case fails, there's nothing else to catch it.