Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 173460 reports (Page 406 of 3470)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 2.73492916s FAIL (2025-09-16 00:13:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.734962666s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 218.187248ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:13:45 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 218.254841ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.209129171s FAIL (2025-09-16 00:13:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.209225879s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.447186096s FAIL (2025-09-16 00:13:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.447212472s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 370.333262ms FAIL (2025-09-16 00:13:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 370.362791ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......XXX...................................................] duration: 2.924130218s FAIL (2025-09-16 00:13:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.924177234s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.908924696s FAIL (2025-09-16 00:13:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.908955621s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 317.087064ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:13:05 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 317.114852ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 206.531296ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:12:53 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 206.553865ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 2.924174872s FAIL (2025-09-16 00:12:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.924213904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XXX..................................................] duration: 2.43498571s FAIL (2025-09-16 00:12:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.43502066s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 323.706299ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:12:36 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 323.746551ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 335.238061ms FAIL (2025-09-16 00:12:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 335.267076ms edge authservice http us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 3.015563867s FAIL (2025-09-16 00:12:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.015856614s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.206089536s FAIL (2025-09-16 00:12:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.206132094s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........XXX.................................................] duration: 3.325933969s FAIL (2025-09-16 00:12:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 3.326034137s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 190.475839ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:12:00 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 190.509026ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.005690438s FAIL (2025-09-16 00:11:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 3.005715261s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 351.520034ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:11:48 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 351.54619ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 331.240538ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:11:20 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 331.382863ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 330.995562ms FAIL (2025-09-16 00:11:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 331.038903ms edge authservice http us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 222.903045ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:11:18 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 222.929412ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.090447526s FAIL (2025-09-16 00:11:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.09048085s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 2.863228462s FAIL (2025-09-16 00:11:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.863256218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........XXX................................................] duration: 2.651419945s FAIL (2025-09-16 00:11:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.651458645s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 337.31677ms FAIL (2025-09-16 00:10:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 337.34405ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.621820702s FAIL (2025-09-16 00:10:44 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.621847222s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 211.33113ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:10:23 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 211.356203ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 360.092362ms FAIL (2025-09-16 00:10:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 360.124542ms edge authservice http us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 325.999826ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:10:14 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 326.091091ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........XXX...............................................] duration: 4.241640832s FAIL (2025-09-16 00:10:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 4.241672651s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.279371008s FAIL (2025-09-16 00:10:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.279402289s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 3.460902269s FAIL (2025-09-16 00:10:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.460929881s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.918400123s FAIL (2025-09-16 00:10:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.919591338s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [..................XX........................................] duration: 218.656507ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:09:59 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 218.679922ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 325.117919ms FAIL (2025-09-16 00:09:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 325.160797ms edge authservice http us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 200.275067ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:09:27 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 200.304206ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.272100956s FAIL (2025-09-16 00:09:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.272138002s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XXX..............................................] duration: 2.576344964s FAIL (2025-09-16 00:09:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.576388704s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 330.459272ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:09:04 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 330.558151ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 3.055999781s FAIL (2025-09-16 00:09:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 3.056096023s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 2.655614627s FAIL (2025-09-16 00:08:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.655635956s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.099710091s FAIL (2025-09-16 00:08:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.099755752s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 356.595304ms FAIL (2025-09-16 00:08:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 356.650667ms edge authservice http us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 205.100802ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:08:43 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 205.183549ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 2.814005205s FAIL (2025-09-16 00:08:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.814028089s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 438.370718ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:08:06 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 438.394846ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.717121685s FAIL (2025-09-16 00:08:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.717161816s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XXX.............................................] duration: 2.610175479s FAIL (2025-09-16 00:08:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.610208968s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 339.244461ms FAIL (2025-09-16 00:08:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 339.27629ms edge authservice http us-select-1

1
...
404
405
406
407
408
...
3470