pu-provisioned linux container — selective outbound network failure

Symptom

A fresh pu-provisioned NixOS container can reach cache.nixos.org (Fastly CDN) but cannot reach github.com or api.github.com from inside nix flake fetches. Every nix invocation that needs to resolve a github:owner/repo flake-ref or download an archive from github.com hangs for 15 s and then errors out. cache.nixos.org pulls in the same nix process succeed normally.

Repro

pu create kolu-pr-958-retry
CI=true nix run github:juspay/justci/lofty-wing -- run --host x86_64-linux=kolu-pr-958-retry

Inside the container, two recipes happened to need GitHub:

# ci/mod.just
ci::nix:
    nix build github:srid/devour-flake -L --no-link --print-out-paths --override-input flake .

ci::pnpm-hash-fresh:
    nix build .#pnpmDeps .#website-pnpm-deps --no-link
    nix build --rebuild .#pnpmDeps .#website-pnpm-deps --no-link

ci::pnpm-hash-fresh resolves a github:nixos/nixpkgs/<rev> input. ci::nix resolves github:srid/devour-flake. Both hang on the same TCP layer.

Container name: kolu-pr-958-retry (now destroyed; the same container also failed on prior attempts).

What works

cache.nixos.org substituter pulls land fine. Excerpt from the same run, same recipe, same nix invocation:

[ci::nix@x86_64-linux] these 7 paths will be fetched (11.14 MiB download, 51.00 MiB unpacked):
[ci::nix@x86_64-linux]   /nix/store/si4q3zks5mn5jhzzyri9hhd3cv789vlm-gcc-15.2.0-lib
[ci::nix@x86_64-linux]   /nix/store/wrxyd3k2f4bmh52pr5rpdjxxsm5r2qxm-gcc-15.2.0-libgcc
[ci::nix@x86_64-linux]   /nix/store/fjkx1l5cnskzrqacf08z7i8z17256w0j-glibc-2.42-61
[ci::nix@x86_64-linux]   /nix/store/vr4agmy8jw7f8kqynpizagdaqxy0ayw4-just-1.50.0
[ci::nix@x86_64-linux]   /nix/store/sgswwrxkhdlfskklqp4gsbi2cskfg07c-libidn2-2.3.8
[ci::nix@x86_64-linux]   /nix/store/i4gg1f526vl5psg5nqniflj4v77vc1kd-libunistring-1.4.2
[ci::nix@x86_64-linux]   /nix/store/xx0z77494lfxr8qjwpck246fry05n3nm-xgcc-15.2.0-libgcc
[ci::nix@x86_64-linux] copying path '/nix/store/wrxyd3k2f4bmh52pr5rpdjxxsm5r2qxm-gcc-15.2.0-libgcc' from 'https://cache.nixos.org' to 'local-overlay://'...
[ci::nix@x86_64-linux] copying path '/nix/store/i4gg1f526vl5psg5nqniflj4v77vc1kd-libunistring-1.4.2' from 'https://cache.nixos.org' to 'local-overlay://'...
[ci::nix@x86_64-linux] copying path '/nix/store/xx0z77494lfxr8qjwpck246fry05n3nm-xgcc-15.2.0-libgcc' from 'https://cache.nixos.org' to 'local-overlay://'...
[ci::nix@x86_64-linux] copying path '/nix/store/fjkx1l5cnskzrqacf08z7i8z17256w0j-glibc-2.42-61' from 'https://cache.nixos.org' to 'local-overlay://'...

i.e. HTTPS egress to cache.nixos.org works; the failure is host-specific.

What fails

Every fetch to github.com or api.github.com hits Timeout was reached (28) Connection timed out after 15000 milliseconds. Nix retries with exponential backoff (342 ms → 586 ms → 1.27 s → 2.49 s → 2.77 s → …) and eventually gives up.

[ci::pnpm-hash-fresh@x86_64-linux] unpacking 'https://github.com/nixos/nixpkgs/archive/f8573b9c935cfaa162dd62cc9e75ae2db86f85df.tar.gz' into the Git cache...
[ci::pnpm-hash-fresh@x86_64-linux] warning: error: unable to download 'https://github.com/nixos/nixpkgs/archive/f8573b9c935cfaa162dd62cc9e75ae2db86f85df.tar.gz': Timeout was reached (28) Connection timed out after 15000 milliseconds; retrying in 342 ms
[ci::nix@x86_64-linux] warning: error: unable to download 'https://api.github.com/repos/srid/devour-flake/commits/HEAD': Timeout was reached (28) Connection timed out after 15000 milliseconds; retrying in 265 ms
[ci::nix@x86_64-linux] warning: error: unable to download 'https://api.github.com/repos/srid/devour-flake/commits/HEAD': Timeout was reached (28) Connection timed out after 15001 milliseconds; retrying in 598 ms
[ci::pnpm-hash-fresh@x86_64-linux] warning: error: unable to download 'https://github.com/nixos/nixpkgs/archive/f8573b9c935cfaa162dd62cc9e75ae2db86f85df.tar.gz': Timeout was reached (28) Connection timed out after 15000 milliseconds; retrying in 586 ms
[ci::nix@x86_64-linux] warning: error: unable to download 'https://api.github.com/repos/srid/devour-flake/commits/HEAD': Timeout was reached (28) Connection timed out after 15000 milliseconds; retrying in 1267 ms
[ci::pnpm-hash-fresh@x86_64-linux] warning: error: unable to download 'https://github.com/nixos/nixpkgs/archive/f8573b9c935cfaa162dd62cc9e75ae2db86f85df.tar.gz': Timeout was reached (28) Connection timed out after 15001 milliseconds; retrying in 1274 ms
[ci::nix@x86_64-linux] warning: error: unable to download 'https://api.github.com/repos/srid/devour-flake/commits/HEAD': Timeout was reached (28) Connection timed out after 15000 milliseconds; retrying in 2487 ms
[ci::pnpm-hash-fresh@x86_64-linux] warning: error: unable to download 'https://github.com/nixos/nixpkgs/archive/f8573b9c935cfaa162dd62cc9e75ae2db86f85df.tar.gz': Timeout was reached (28) Connection timed out after 15001 milliseconds; retrying in 2771 ms
[ci::nix@x86_64-linux] error:
[ci::nix@x86_64-linux]        error: unable to download 'https://api.github.com/repos/srid/devour-flake/commits/HEAD': Timeout was reached (28) Connection timed out after 15000 milliseconds
[ci::nix@x86_64-linux] error: Recipe `nix` failed on line 64 with exit code 1
[ci::pnpm-hash-fresh@x86_64-linux] error:
[ci::pnpm-hash-fresh@x86_64-linux]        error: Failed to open archive (Source threw exception: error: unable to download 'https://github.com/nixos/nixpkgs/archive/f8573b9c935cfaa162dd62cc9e75ae2db86f85df.tar.gz': Timeout was reached (28) Connection timed out after 15000 milliseconds)
[ci::pnpm-hash-fresh@x86_64-linux] error: Recipe `pnpm-hash-fresh` failed on line 59 with exit code 1

Error code 28 from libcurl = CURLE_OPERATION_TIMEDOUT (TCP connect / TLS handshake / read all eligible). Combined with cache.nixos.org working in the same process, this looks like a selective block / filtered route to GitHub-owned IPs, not blanket no-internet.

Hypotheses for the pu admin

1. Outbound egress filter that whitelists cache.nixos.org (or its Fastly CDN ASN) but not github.com / api.github.com.
2. Tailscale / SOCKS proxy chain routing *.nixos.org differently from github.com. The host runs proxychains (visible in the host's pu CLI output as [proxychains] DLL init); if the container's HTTPS egress traverses a proxy, GitHub's TLS SNI may be getting dropped while Fastly's passes.
3. IPv6-only path to GitHub failing if the container has no v6 (GitHub recently expanded AAAA records — some networks hit "AAAA preferred, A unreachable" timeouts at exactly 15 s).
4. MTU mismatch on the tunnel — github.com returns larger TLS records than cache.nixos.org and triggers PMTUD black-holing. cache.nixos.org works because the response payload fits in the first segment.

Confirming via:

pu connect kolu-pr-958-retry -- 'curl -v --max-time 20 https://api.github.com/'
pu connect kolu-pr-958-retry -- 'curl -v --max-time 20 https://cache.nixos.org/nix-cache-info'
pu connect kolu-pr-958-retry -- 'getent hosts github.com api.github.com cache.nixos.org'
pu connect kolu-pr-958-retry -- 'ip -6 route show; ip -4 route show'

would distinguish 1/2 from 3/4 directly.

Blast radius

Every justci pipeline node depends on ci::nix (which fetches srid/devour-flake from GitHub) and ci::pnpm-hash-fresh (which evaluates a github:nixos/nixpkgs input). When both fail, the rest of the DAG cascade-skips with Skipped (upstream failed). Net effect: the entire linux side of a kolu CI run is unrunnable from within pu-provisioned containers, even though the same flake builds cleanly on the static darwin runner and on developer laptops.

Workaround currently in use

None — kolu's .agency/do.md calls pu create per CI invocation. As a temporary measure, just ci::e2e runs locally outside the container (349/349 scenarios pass), but that loses the per-run isolation the ephemeral container provides.

Container details

• Provisioned via pu create kolu-pr-958-retry
• NixOS (uname not captured at the time; can re-provision and run uname -a; nixos-version if useful)
• Storage volume name: pu-home-kolu-pr-958-retry (deleted at end of run)