Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 354031 reports (Page 561 of 7081)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 48 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXXXXXXXXXX] duration: 5.000788662s trace: https://jaeger.dev.storj.io/trace/16b070b118a9a87f FAIL (2026-03-18 21:12:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000812454s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXXXXXXXXX] duration: 5.000859299s trace: https://jaeger.dev.storj.io/trace/1d167bd19f51ce96 FAIL (2026-03-18 21:11:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000885294s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 176.27985ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:10:47 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 176.305443ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXXXXXXXX] duration: 5.001805456s trace: https://jaeger.dev.storj.io/trace/5dedeb4471048d1e FAIL (2026-03-18 21:10:23 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001837253s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 166.082787ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:09:51 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 166.107157ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 163.02574ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:09:35 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 163.053553ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXXXXXXX] duration: 5.001507066s trace: https://jaeger.dev.storj.io/trace/10d00e340acafc51 FAIL (2026-03-18 21:09:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001692411s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 176.827593ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:09:03 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 176.863465ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 162.842842ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:08:41 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 162.862599ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.....XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXXXXXX] duration: 5.000711964s trace: https://jaeger.dev.storj.io/trace/656dd0981df43b80 FAIL (2026-03-18 21:08:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000739137s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 164.156032ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:08:03 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 164.187714ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXXXXX] duration: 5.000959236s trace: https://jaeger.dev.storj.io/trace/1029deda2e7bc986 FAIL (2026-03-18 21:07:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000981602s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 167.542938ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:07:05 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 167.571827ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [.......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXXXX] duration: 5.001226238s trace: https://jaeger.dev.storj.io/trace/3fe18778859f8718 FAIL (2026-03-18 21:06:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00125743s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 169.379898ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:05:59 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 169.433476ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXXX] duration: 5.001667574s trace: https://jaeger.dev.storj.io/trace/5d2c376bd9f5f7ba FAIL (2026-03-18 21:05:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001699812s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 160.93494ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:05:29 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 160.96873ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXX] duration: 5.001120062s trace: https://jaeger.dev.storj.io/trace/29cbfba23cda9344 FAIL (2026-03-18 21:05:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001147148s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXX] duration: 5.006113832s trace: https://jaeger.dev.storj.io/trace/2ff8d9c4e925b8fe FAIL (2026-03-18 21:04:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.006144717s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 160.967463ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:04:29 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 161.005443ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXX] duration: 5.00176788s trace: https://jaeger.dev.storj.io/trace/50f4939ef0da3f5 FAIL (2026-03-18 21:03:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001793021s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 178.898486ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:03:47 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 178.923623ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXX] duration: 5.001107875s trace: https://jaeger.dev.storj.io/trace/5802337a7c5adf98 FAIL (2026-03-18 21:03:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001132406s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 166.408424ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:02:45 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 166.437154ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XX] duration: 5.027141022s trace: https://jaeger.dev.storj.io/trace/5f6223fbe931beaa FAIL (2026-03-18 21:02:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.027243041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 8.004291704s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 21:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.004320222s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 181.806902ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:01:42 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 181.827568ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........X] duration: 5.001244018s trace: https://jaeger.dev.storj.io/trace/1b268f7f34e0dfc2 FAIL (2026-03-18 21:01:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001266455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 7.407566271s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 21:01:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.407598755s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 179.39254ms remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:01:04 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 179.419782ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........] duration: 3.588830447s trace: https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8 FAIL (2026-03-18 21:00:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588871425s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 10.151382403s remote: "tcp:[2620:cb:2000::1]:443" response headers: Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10 FAIL (2026-03-18 21:00:20 UTC) orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n" Duration: 10.151411607s core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 3.29743133s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 21:00:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.297465803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 2.350863584s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:59:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.350955446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.XXX.......................................................] duration: 2.658503626s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:59:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.658529983s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [.X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.........] duration: 1.957329032s trace: https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8 FAIL (2026-03-18 20:59:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.957351662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX.XXX......................................................] duration: 2.370284399s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:58:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.370445622s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [..X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX........] duration: 2.660822224s trace: https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8 FAIL (2026-03-18 20:58:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.660862185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [XXX.XXX.....................................................] duration: 5.355621978s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:57:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.355651426s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [...X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.......] duration: 4.354626093s trace: https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8 FAIL (2026-03-18 20:57:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.354730587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XXXX.XXX....................................................] duration: 4.804301656s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:56:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.80433413s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX......] duration: 4.214057776s trace: https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8 FAIL (2026-03-18 20:56:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.214097992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXXXX.XXX...................................................] duration: 5.901584407s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:56:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.901615459s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [X....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.....] duration: 4.457305018s trace: https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8 FAIL (2026-03-18 20:55:54 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.457338251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXXXXX.XXX..................................................] duration: 5.224205486s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:55:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.224243864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX....] duration: 4.371193792s trace: https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8 FAIL (2026-03-18 20:55:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.371225811s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXXX.XXX.................................................] duration: 5.609203484s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:55:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.609228643s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXXX.XXX................................................] duration: 5.591279674s trace: https://jaeger.dev.storj.io/trace/701168f22f72988f FAIL (2026-03-18 20:54:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments... Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment... Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment. Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece... Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments... Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil> Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments... Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.591321855s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...] duration: 4.289656751s trace: https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8 FAIL (2026-03-18 20:54:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.28969689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..] duration: 5.001157105s trace: https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8 FAIL (2026-03-18 20:54:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001181301s core uplink us-select-1 us1-placement-20

1
...
559
560
561
562
563
...
7081