Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 174236 reports (Page 424 of 3485)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 582.494277ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:00:07 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 582.723719ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 370.661814ms FAIL (2025-09-15 23:59:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 370.713677ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.936879595s FAIL (2025-09-15 23:59:49 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.936917083s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................XXX...............................] duration: 2.859402319s FAIL (2025-09-15 23:59:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.859434014s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 3.505076845s FAIL (2025-09-15 23:59:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.505108s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 199.989463ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:59:42 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 200.018598ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.12116364s FAIL (2025-09-15 23:59:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.121187596s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 359.288564ms FAIL (2025-09-15 23:59:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 359.314281ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 323.29325ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:59:23 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 323.321476ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................XXX..............................] duration: 2.60838656s FAIL (2025-09-15 23:59:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.608418928s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 3.09482023s FAIL (2025-09-15 23:59:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 3.094849802s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................XXX.............................] duration: 2.776328674s FAIL (2025-09-15 23:58:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.776367794s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 210.412185ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:58:58 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 210.443441ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 2.896235291s FAIL (2025-09-15 23:58:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.896259929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................XXX............................] duration: 2.588347768s FAIL (2025-09-15 23:58:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.58838298s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.337537109s FAIL (2025-09-15 23:58:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.33756731s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 336.821831ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:58:28 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 336.851341ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 199.520234ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:58:27 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 199.54759ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.694788017s FAIL (2025-09-15 23:58:26 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.694815618s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 333.813391ms FAIL (2025-09-15 23:58:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 333.879174ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.438214982s FAIL (2025-09-15 23:58:12 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.438236746s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................XXX...........................] duration: 2.522153622s FAIL (2025-09-15 23:58:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.522219911s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 316.232414ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:58:05 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 316.333983ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.131825342s FAIL (2025-09-15 23:57:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.131854281s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 360.847116ms FAIL (2025-09-15 23:57:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 360.937349ms edge authservice http us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 214.474181ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:57:55 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 214.502786ms edge linksharing us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 322.881419ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:57:40 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 322.913357ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 2.802837454s FAIL (2025-09-15 23:57:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.802858089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................XXX..........................] duration: 2.67124952s FAIL (2025-09-15 23:57:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.671264306s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.222252843s FAIL (2025-09-15 23:57:38 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 3.222296239s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.084612773s FAIL (2025-09-15 23:57:38 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.084637844s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 212.004176ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:57:35 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 212.02679ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 394.508611ms FAIL (2025-09-15 23:57:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 394.554722ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 2.707186858s FAIL (2025-09-15 23:57:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.707777842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XXX.........................] duration: 2.896279437s FAIL (2025-09-15 23:57:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.896309993s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 196.370456ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:57:04 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 196.404783ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.951041922s FAIL (2025-09-15 23:56:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.951068959s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 342.489523ms FAIL (2025-09-15 23:56:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 342.531212ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 311.502322ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:56:45 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 311.52993ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.736173843s FAIL (2025-09-15 23:56:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.736219437s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 199.150742ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:56:32 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 199.178028ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 3.095951037s FAIL (2025-09-15 23:56:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.095976054s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 378.599381ms FAIL (2025-09-15 23:56:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 378.628369ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.169435562s FAIL (2025-09-15 23:56:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.169480441s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 469.79179ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:56:22 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 469.82018ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................XXX........................] duration: 3.352273683s FAIL (2025-09-15 23:56:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 3.352312851s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.883124505s FAIL (2025-09-15 23:56:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.883193277s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 215.139605ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:55:55 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 215.165979ms edge linksharing us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.................................XX.........................] duration: 297.51344ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:55:53 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 297.564546ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................XXX.......................] duration: 2.430367044s FAIL (2025-09-15 23:55:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.430427227s core uplink us-select-1

1
...
422
423
424
425
426
...
3485