Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 173460 reports (Page 410 of 3470)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.579784355s FAIL (2025-09-15 23:52:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.579820256s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................XXX...................] duration: 2.529300984s FAIL (2025-09-15 23:52:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.529372554s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 318.031048ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:52:40 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 318.062796ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 204.007331ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:52:29 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 204.043789ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 3.12322523s FAIL (2025-09-15 23:52:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.123295607s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 380.097714ms FAIL (2025-09-15 23:52:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 380.130427ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XXX..................] duration: 2.659588523s FAIL (2025-09-15 23:52:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.659695357s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.809482473s FAIL (2025-09-15 23:52:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.809508455s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 340.823541ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:51:52 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 340.850903ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 408.65856ms FAIL (2025-09-15 23:51:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 408.70144ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 3.251528371s FAIL (2025-09-15 23:51:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.251549123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.113883488s FAIL (2025-09-15 23:51:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.113905389s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................XXX.................] duration: 2.548538155s FAIL (2025-09-15 23:51:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.548573965s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 344.616316ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:51:34 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 344.64724ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 201.006735ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:51:26 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 201.039868ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 3.073359373s FAIL (2025-09-15 23:51:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.073382895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.392379644s FAIL (2025-09-15 23:51:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.392409492s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................XXX................] duration: 3.037236752s FAIL (2025-09-15 23:51:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 3.037267493s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 336.244777ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:51:01 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 336.280749ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.945779642s FAIL (2025-09-15 23:51:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.945814504s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................XXX...............] duration: 3.771919575s FAIL (2025-09-15 23:50:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 3.771955685s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 3.121059962s FAIL (2025-09-15 23:50:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.121102788s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 228.263678ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:50:46 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 228.291886ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 332.545847ms FAIL (2025-09-15 23:50:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 332.582726ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 336.466328ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:50:32 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 336.49622ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 2.903714379s FAIL (2025-09-15 23:50:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.903740323s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 302.380307ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:50:11 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 302.40601ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.702996334s FAIL (2025-09-15 23:50:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.703028905s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................XXX..............] duration: 2.933187368s FAIL (2025-09-15 23:50:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.933220237s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.185913794s FAIL (2025-09-15 23:50:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.185963761s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 398.473763ms FAIL (2025-09-15 23:50:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 398.66613ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [............................................XX..............] duration: 330.020469ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:49:54 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 330.04706ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 229.015857ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:49:39 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 229.043934ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................XXX.............] duration: 2.683394625s FAIL (2025-09-15 23:49:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.68343882s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 3.00071961s FAIL (2025-09-15 23:49:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.000819707s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 355.298488ms FAIL (2025-09-15 23:49:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 355.336233ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.668847796s FAIL (2025-09-15 23:49:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.668876264s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 324.978984ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:49:06 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 325.007414ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.195454728s FAIL (2025-09-15 23:49:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.195487765s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 213.406181ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:48:55 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 213.428108ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................XXX............] duration: 2.905181734s FAIL (2025-09-15 23:48:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.905216378s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.817671033s FAIL (2025-09-15 23:48:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.817713107s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 235.450548ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:48:26 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 235.480063ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 368.571876ms FAIL (2025-09-15 23:48:26 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 368.602399ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 360.63375ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:48:20 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 360.663588ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 1.104420368s FAIL (2025-09-15 23:48:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.104463321s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration: 2.566765046s FAIL (2025-09-15 23:48:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.566850447s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 479.369995ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-15 23:48:02 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 479.388617ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 3.136629257s FAIL (2025-09-15 23:47:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.136658696s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 206.841103ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-15 23:47:36 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 206.868193ms edge linksharing us-select-1

1
...
408
409
410
411
412
...
3470