Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.......X.......] duration:
169.379898ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10
FAIL (2026-03-18 21:05:59 UTC)
orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n"
Duration: 169.433476ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXXX] duration:
5.001667574s trace:
https://jaeger.dev.storj.io/trace/5d2c376bd9f5f7ba
FAIL (2026-03-18 21:05:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001699812s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [........X......] duration:
160.93494ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10
FAIL (2026-03-18 21:05:29 UTC)
orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n"
Duration: 160.96873ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [.........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXXX] duration:
5.001120062s trace:
https://jaeger.dev.storj.io/trace/29cbfba23cda9344
FAIL (2026-03-18 21:05:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001147148s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [..........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXXX] duration:
5.006113832s trace:
https://jaeger.dev.storj.io/trace/2ff8d9c4e925b8fe
FAIL (2026-03-18 21:04:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.006144717s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.........X.....] duration:
160.967463ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10
FAIL (2026-03-18 21:04:29 UTC)
orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n"
Duration: 161.005443ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [...........XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXXX] duration:
5.00176788s trace:
https://jaeger.dev.storj.io/trace/50f4939ef0da3f5
FAIL (2026-03-18 21:03:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001793021s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [..........X....] duration:
178.898486ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10
FAIL (2026-03-18 21:03:47 UTC)
orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n"
Duration: 178.923623ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XXX] duration:
5.001107875s trace:
https://jaeger.dev.storj.io/trace/5802337a7c5adf98
FAIL (2026-03-18 21:03:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001132406s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [...........X...] duration:
166.408424ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10
FAIL (2026-03-18 21:02:45 UTC)
orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n"
Duration: 166.437154ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [.............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........XX] duration:
5.027141022s trace:
https://jaeger.dev.storj.io/trace/5f6223fbe931beaa
FAIL (2026-03-18 21:02:12 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.027243041s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
8.004291704s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 21:02:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 8.004320222s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [............X..] duration:
181.806902ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10
FAIL (2026-03-18 21:01:42 UTC)
orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n"
Duration: 181.827568ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........X] duration:
5.001244018s trace:
https://jaeger.dev.storj.io/trace/1b268f7f34e0dfc2
FAIL (2026-03-18 21:01:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001266455s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
7.407566271s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 21:01:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 7.407598755s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [.............X.] duration:
179.39254ms remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10
FAIL (2026-03-18 21:01:04 UTC)
orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n"
Duration: 179.419782ms
core
website
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..........] duration:
3.588830447s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 21:00:32 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.588871425s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)antiblip:
1 out of 15 is failed (threshold=6) [..............X] duration:
10.151382403s remote:
"tcp:[2620:cb:2000::1]:443" response headers:
Date=Wed, 18 Mar 2026 21:00:20 GMT Content-Type=text/html Set-Cookie=_cfuvid=0E8f4XRQWuNoX39W5W9HX68mp5cvkr0xvI.tzVww3io-1773867610.3114054-1.0.1.1-0u9ck4Xw2OZxxE_tJAvzWJhkEolF0JoEoWZbPLUdaFc; HttpOnly; SameSite=None; Secure; Path=/; Domain=www.storj.io Server=cloudflare Vary=Accept-Encoding Content-Length=132 Cf-Ray=9de72a547ccfdba5-FRA Cf-Cache-Status=BYPASS Alt-Svc=h3=":443"; ma=86400 target:
https://www.storj.io/dcs-satellites trace:
https://jaeger.dev.storj.io/trace/3a2f5b3f6566f10
FAIL (2026-03-18 21:00:20 UTC)
orbiter warning: unexpected status code "504 Gateway Timeout" "<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n</body>\r\n</html>\r\n"
Duration: 10.151411607s
core
website
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
3.29743133s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 21:00:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 3.297465803s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.XXX........................................................] duration:
2.350863584s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:59:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.350955446s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X.XXX.......................................................] duration:
2.658503626s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:59:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.658529983s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [.X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.........] duration:
1.957329032s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:59:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.957351662s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [XX.XXX......................................................] duration:
2.370284399s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:58:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.370445622s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [..X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX........] duration:
2.660822224s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:58:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.660862185s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [XXX.XXX.....................................................] duration:
5.355621978s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:57:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.355651426s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [...X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.......] duration:
4.354626093s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:57:34 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.354730587s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [XXXX.XXX....................................................] duration:
4.804301656s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:56:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 4.80433413s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX......] duration:
4.214057776s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:56:24 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.214097992s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [XXXXX.XXX...................................................] duration:
5.901584407s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:56:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.901615459s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [X....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.....] duration:
4.457305018s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:55:54 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.457338251s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [XXXXXX.XXX..................................................] duration:
5.224205486s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:55:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.224243864s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [XX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX....] duration:
4.371193792s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:55:31 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.371225811s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [XXXXXXX.XXX.................................................] duration:
5.609203484s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:55:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.609228643s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [XXXXXXXX.XXX................................................] duration:
5.591279674s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:54:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.591321855s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX...] duration:
4.289656751s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:54:37 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.28969689s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX..] duration:
5.001157105s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:54:17 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001181301s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [XXXXXXXXX.XXX...............................................] duration:
6.049422836s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:53:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.049451881s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [X.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX.] duration:
4.071101728s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:53:46 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.071169745s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXX..............................................] duration:
4.916407341s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:53:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 4.916434153s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [XX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration:
5.000839926s trace:
https://jaeger.dev.storj.io/trace/78133c0a1c14a8d8
FAIL (2026-03-18 20:52:55 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000863533s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXX.............................................] duration:
5.438291738s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:52:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.438324645s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXX............................................] duration:
5.202178502s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:52:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.202206854s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [XXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration:
5.001470759s trace:
https://jaeger.dev.storj.io/trace/4edfc4c630d51d83
FAIL (2026-03-18 20:51:56 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001495621s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [XXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
4.109717422s trace:
https://jaeger.dev.storj.io/trace/1f7cb97b10ff47b6
FAIL (2026-03-18 20:51:39 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.109749477s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXX...........................................] duration:
4.927591819s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:51:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 4.927619811s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [XXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001648208s trace:
https://jaeger.dev.storj.io/trace/1f7cb97b10ff47b6
FAIL (2026-03-18 20:50:56 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001675095s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXX..........................................] duration:
4.924818675s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:50:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 4.924846764s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [XXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00157905s trace:
https://jaeger.dev.storj.io/trace/729335be941ed8fa
FAIL (2026-03-18 20:50:17 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001604633s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXX.........................................] duration:
7.001901736s trace:
https://jaeger.dev.storj.io/trace/701168f22f72988f
FAIL (2026-03-18 20:49:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:17:12.361144942 (196h6m15.376814543s) [upload=188367]: Starting upload
Mar 18 20:17:12.361206137 (196h6m15.376875746s) [upload=188367]: Done starting upload
Mar 18 20:17:12.361572063 (196h6m15.377241653s) [upload=188367]: Uploading segments...
Mar 18 20:17:12.362109301 (196h6m15.377778901s) [upload=188367]: Got next segment. Inline: false
Mar 18 20:17:12.362165253 (196h6m15.377834843s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:17:12.362170553 (196h6m15.377840153s) [upload=188367]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:17:12.481954728 (196h6m15.497624328s) [seg_pos={0 0}, upload=188367]: Begin upload segment...
Mar 18 20:17:12.482033402 (196h6m15.497703003s) [seg_pos={0 0}, upload=188367]: Done begin upload segment.
Mar 18 20:17:12.482069519 (196h6m15.497739135s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482111964 (196h6m15.497781566s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.482797548 (196h6m15.498467151s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483403786 (196h6m15.499073388s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483833779 (196h6m15.499503374s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.483984371 (196h6m15.499653979s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484430456 (196h6m15.500100058s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.484813905 (196h6m15.500483515s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485022737 (196h6m15.500692348s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.485804661 (196h6m15.501474268s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.486490562 (196h6m15.502160174s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.491363882 (196h6m15.507033473s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.493048663 (196h6m15.508718268s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.494902609 (196h6m15.51057222s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496223721 (196h6m15.511893336s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:12.496871294 (196h6m15.512540881s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Uploading piece...
Mar 18 20:17:13.109179904 (196h6m16.124849497s) [upload=188367]: Next returned nil segment
Mar 18 20:17:13.109195792 (196h6m16.124865391s) [upload=188367]: Waiting for error group managing segments...
Mar 18 20:17:13.235977836 (196h6m16.251647446s) [noise=true , address=45.91.116.5:4332 , piece_id=2FVE4N23BWGNZGIXPWPCGTAAV5SF4MIKNUEDZXSSLUNLI6DGJ43Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.236828084 (196h6m16.252497695s) [noise=true , address=45.91.116.13:4156 , piece_id=KVWRZH426V6AYCF7XPAWZO5ICHZKRYKUXIJD6CLROYF5OU7BR4WQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.237906572 (196h6m16.253576185s) [noise=true , address=45.91.116.4:4952 , piece_id=7CWRWMGFMTVW2EVSJ3HF76W63CKLLAXFA75SFZFQ7E5OLTMQQPIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.240724942 (196h6m16.256394543s) [noise=true , address=45.91.116.14:4196 , piece_id=L4YS5UTF6TFNJDVXRVVMJJ26NIXQWD2VX67PT3DZMB43QL3RNR4A, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.241160483 (196h6m16.256830085s) [noise=true , address=77.90.31.21:4878 , piece_id=HGTFZW24XGOL3HS7E7SWWSI4P5IX7R3JSB5JTRUIMRLPMOQJONJA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244200746 (196h6m16.259870363s) [noise=true , address=77.90.31.18:4805 , piece_id=KGY4MGYXIOYAGO62AMRDHRLEYTZXHZHLZ4VM2A3UCUF6MLINHYIA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.244868320 (196h6m16.260537931s) [noise=true , address=77.90.31.17:4773 , piece_id=6XBA3RKCYSA543TCLG5MQDWQDJXYZOIM6AMXUBOLWXHOGFAWDLFQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246480486 (196h6m16.262150097s) [noise=true , address=77.90.31.12:4589 , piece_id=ZQ73JPTQOFR3X5DTQSRZAOFEYJ6W2B2HBL5X5BPV3U2UQQWEECYQ, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.246656657 (196h6m16.262326272s) [noise=true , address=45.91.116.11:4118 , piece_id=7HB233SAWCF4RT4HDXGPILQLWELNM7IKVRSRW4RVVMW5E73A6XOA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.260192886 (196h6m16.27586249s) [noise=true , address=79.127.219.41:4927 , piece_id=VFPKT4A6V2ZZYCYYKDI6YTKO3M64FW7VMJUVD4AXGGZ3EW34BZ4Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.263800730 (196h6m16.279470334s) [noise=true , address=45.91.116.7:4012 , piece_id=J3Q5XD5INBDAKWETGRYNOCEG4QK2LIJ2B766CE3FWFLO3Y5YB47Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:13.296925335 (196h6m16.312594935s) [noise=true , address=79.127.219.45:4854 , piece_id=DMZOHNXKRVKDBTDNLPFWCZXCTW4ESTS7PG2GDVOY2QS42HI37P7Q, seg_pos={0 0}, upload=188367]: Done uploading piece. err: <nil>
Mar 18 20:17:42.079775974 (196h6m45.095445595s) [noise=true , address=145.79.227.42:4123 , piece_id=EWFO5SWHRCPIJNBJNDCJIICDV6OT2EYBNDG7DMVTGY4KEWZEHRUA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, address:145.79.227.42:4123): protocol: rpc: dial tcp 145.79.227.42:4123: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:17:42.079846236 (196h6m45.095515847s) [noise=true , address=145.79.227.30:4765 , piece_id=H6C67TVXOJYQ4LAG4KNLLJPKNNVBZDXJU4LEMX6ITATWFINUQ6CA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:145.79.227.30:4765): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080088588 (196h6m45.0957582s) [noise=true , address=145.79.227.22:4515 , piece_id=HWBAY2576KTM2NTK24KRE7CHXM4YMXZYXY62ZRCDXGFSCOL4C5SA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:1ZZmn2sRUkYuNmHY3RY7skQwAYoVYth7rJ8DhmjEpzgE4Rtmxb, address:145.79.227.22:4515): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080320204 (196h6m45.095989828s) [noise=true , address=145.79.227.31:4801 , piece_id=AASN5YQJ7XSMBS2HCQCVRSVHYEQ4OCGMIPDTBK2IVDVFBMXUUXKA, seg_pos={0 0}, upload=188367]: Done uploading piece. err: ecclient: upload failed (node:12pdMen2HsTBypaNt4cUu12TWjhbKnk3V287JJus1273x28xDXM, address:145.79.227.31:4801): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:17:42.080357972 (196h6m45.096027572s) [seg_pos={0 0}, upload=188367]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:17:42.186561781 (196h6m45.202231372s) [upload=188367]: Done uploading segments...
Mar 18 20:17:42.186597961 (196h6m45.202267563s) [upload=188367]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 7.001937249s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [XXXXXXX.XXX....X..............XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001706823s trace:
https://jaeger.dev.storj.io/trace/396b66121b990631
FAIL (2026-03-18 20:49:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00173138s
core
uplink
us-select-1
us1-placement-20
|