Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 174236 reports (Page 423 of 3485)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [...................XX.......................................] duration: 307.748836ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:05:19 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 307.77542ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XXX........................................] duration: 2.551726576s FAIL (2025-09-16 00:05:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.551836421s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 2.966654295s FAIL (2025-09-16 00:05:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.966762707s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 205.802997ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:04:59 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 205.826469ms edge linksharing us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 376.041695ms FAIL (2025-09-16 00:04:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 376.068643ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.593122862s FAIL (2025-09-16 00:04:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.593668666s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [....................XX......................................] duration: 322.340488ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:04:26 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 322.365285ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 2.843370639s FAIL (2025-09-16 00:04:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.843404586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.731690498s FAIL (2025-09-16 00:04:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.731715216s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.104008245s FAIL (2025-09-16 00:04:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.10453867s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 364.754455ms FAIL (2025-09-16 00:04:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 364.784949ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................XXX.......................................] duration: 2.597426195s FAIL (2025-09-16 00:04:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.597465581s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.....................XX.....................................] duration: 358.930188ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:04:02 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 359.041687ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 201.803781ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:04:00 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 201.849161ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.761307576s FAIL (2025-09-16 00:03:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.761341172s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.458529338s FAIL (2025-09-16 00:03:43 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.458555389s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [..........................XX................................] duration: 206.176235ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:03:30 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 206.218899ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 3.52779813s FAIL (2025-09-16 00:03:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527913332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.854947724s FAIL (2025-09-16 00:03:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.854973954s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [......................XX....................................] duration: 323.30042ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:03:15 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 323.326881ms edge gateway vhost us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 330.243363ms FAIL (2025-09-16 00:03:09 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 330.28255ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................XXX......................................] duration: 3.123545093s FAIL (2025-09-16 00:03:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 3.123581253s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.605439657s FAIL (2025-09-16 00:03:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.605471873s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.833274587s FAIL (2025-09-16 00:02:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.833316635s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................XXX.....................................] duration: 3.236943715s FAIL (2025-09-16 00:02:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 3.236982023s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 338.758554ms FAIL (2025-09-16 00:02:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 338.783201ms edge authservice http us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [...........................XX...............................] duration: 202.963703ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:02:29 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 202.985489ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 3.068746971s FAIL (2025-09-16 00:02:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.068774476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................XXX....................................] duration: 2.866612767s FAIL (2025-09-16 00:02:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.866641767s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.115456359s FAIL (2025-09-16 00:02:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.115479434s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.......................XX...................................] duration: 394.783575ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:02:14 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 394.807373ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [............................XX..............................] duration: 207.497349ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:02:09 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 207.522395ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.7234163s FAIL (2025-09-16 00:02:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.723437614s edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 354.088512ms FAIL (2025-09-16 00:02:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 354.135494ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................XXX...................................] duration: 2.540703694s FAIL (2025-09-16 00:01:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.540733483s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [........................XX..................................] duration: 317.287907ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:01:43 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 317.332594ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...............................XX...........................] duration: 2.800194152s FAIL (2025-09-16 00:01:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.800223371s core uplink us-select-1 us1-placement-20
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 352.545502ms FAIL (2025-09-16 00:01:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 352.649919ms edge authservice http us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.145240503s FAIL (2025-09-16 00:01:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.145266587s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 3.070858907s FAIL (2025-09-16 00:01:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 3.070886912s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [.............................XX.............................] duration: 217.042346ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:01:07 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 217.14133ms edge linksharing us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.........................XX.................................] duration: 384.53556ms target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-09-16 00:01:01 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": dial tcp 45.38.24.2:443: i/o timeout Duration: 384.571657ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................XXX..................................] duration: 2.728513028s FAIL (2025-09-16 00:01:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.728555967s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 386.489102ms FAIL (2025-09-16 00:00:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": dial tcp 45.38.24.2:443: i/o timeout Duration: 386.527963ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................XXX.................................] duration: 2.988326682s FAIL (2025-09-16 00:00:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.988360166s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [................................XX..........................] duration: 2.889536783s FAIL (2025-09-16 00:00:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.889583566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.556721961s FAIL (2025-09-16 00:00:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 1.556741602s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................XXX................................] duration: 2.91974887s FAIL (2025-09-16 00:00:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 15 23:40:26.070475751 (417h32m46.562908204s) [upload=367243]: Starting upload Sep 15 23:40:26.070566142 (417h32m46.562998574s) [upload=367243]: Done starting upload Sep 15 23:40:26.070793734 (417h32m46.563226168s) [upload=367243]: Uploading segments... Sep 15 23:40:26.071016487 (417h32m46.563448921s) [upload=367243]: Got next segment. Inline: false Sep 15 23:40:26.071063323 (417h32m46.563495775s) [upload=367243]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 15 23:40:26.071076339 (417h32m46.563508792s) [upload=367243]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 15 23:40:26.184942850 (417h32m46.677375304s) [seg_pos={0 0}, upload=367243]: Begin upload segment... Sep 15 23:40:26.185028069 (417h32m46.677460517s) [seg_pos={0 0}, upload=367243]: Done begin upload segment. Sep 15 23:40:26.185068989 (417h32m46.677501441s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.185724240 (417h32m46.678156692s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186057058 (417h32m46.678489511s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186314850 (417h32m46.678747303s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186491297 (417h32m46.678923748s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186897946 (417h32m46.679330395s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.186916513 (417h32m46.679348962s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187216141 (417h32m46.679648585s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187410059 (417h32m46.67984251s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187548554 (417h32m46.679981007s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187891239 (417h32m46.680323691s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.187930010 (417h32m46.680362462s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188300671 (417h32m46.680733122s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188597680 (417h32m46.681030122s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.188939904 (417h32m46.681372353s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.189187119 (417h32m46.681619564s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Uploading piece... Sep 15 23:40:26.808170182 (417h32m47.300602626s) [upload=367243]: Next returned nil segment Sep 15 23:40:26.808216420 (417h32m47.300648865s) [upload=367243]: Waiting for error group managing segments... Sep 15 23:40:26.946199863 (417h32m47.438632293s) [noise=true , address=45.91.116.11:4118 , piece_id=BBDOSLC5DCETT7MVATBJH6MGBEJ3J5GV7I5EU3X7F4Y7L7A7N3WQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.946330956 (417h32m47.438763403s) [noise=true , address=45.91.116.7:4009 , piece_id=FJUXJA3SCEHKZQ3TF7XAHBONPENSHA2EZM5VPOUWT5AF5FFAWBXA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.958202290 (417h32m47.450634723s) [noise=true , address=45.91.116.6:4973 , piece_id=6CEAU6FYIJ3OFB5GR57SVP45CJHKTYCFQIL5YPYQ6CGUCKB53CVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.961831650 (417h32m47.454264082s) [noise=true , address=45.91.116.10:4083 , piece_id=6QPRCJSSBIIVFNULJGE2BSONQMHL4ZSBXMEXMISXVV2KCVHJBX3A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.962849577 (417h32m47.455282006s) [noise=true , address=45.91.116.6:4974 , piece_id=WJL5MLTVRLGWJLYDYOLNXZ2EFOEVOVSMFUYWHKOA3TTWZLKWQUSA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:26.969890777 (417h32m47.46232321s) [noise=true , address=45.91.116.8:4048 , piece_id=BYJVXDVFY52FDQYJ7NWQCQ3KASG5ZLSTO66DNYZGT3SPYGXC2YVA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.005327018 (417h32m47.497759469s) [noise=true , address=45.91.116.8:4047 , piece_id=FTL6SUC6QJGQL6FSE7QVDQOOOBMA6HO2URKVQBQD5WW6ON5LM6CA, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.050024747 (417h32m47.542457189s) [noise=true , address=145.79.227.15:4873 , piece_id=WDMWNGRHW3BXXT54GZOIVGD3EICENKQ2PHNV3LMXWZIBNCOUM2VQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.065442548 (417h32m47.557874977s) [noise=true , address=145.79.227.11:4730 , piece_id=JNN7RLZF3OTHAFVIYPFUTOEDREQ3CNN5LXIUFGAD4XNKQEUQIWRQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:27.166046867 (417h32m47.658479304s) [noise=true , address=145.79.227.4:4185 , piece_id=IXUXFHQSDIBJ3K3LR26ERNKEA2QXZVU5QH3GKZ65NOKWVBBNRB6A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: <nil> Sep 15 23:40:55.784307815 (417h33m16.276740265s) [noise=true , address=77.90.31.10:4518 , piece_id=CCDYRFIWKABBJPOCBWPIGEQQYSHNBLKEOQOPXQE5OFSGT6VXCEGQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 139.59.144.22:51944->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784436862 (417h33m16.276869296s) [noise=true , address=77.90.31.10:4520 , piece_id=RZNYFFC4AVVFRKDRKVAVAPX2JNUNIMOCWCVG3L3LETHT4HZEF4CQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784539779 (417h33m16.276972222s) [noise=true , address=77.90.31.17:4772 , piece_id=76ORLKXHG3WEWRSLLOHQW3RRGVU5GYG3IEJGHNMWLMPZYS7MSFUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784719697 (417h33m16.277152133s) [noise=true , address=77.90.31.12:4592 , piece_id=CFBKUEALXCLOEQLPUVBOK4RHCFXFJV5XZHEIPOTFBZIKGLKXYP2A, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784747558 (417h33m16.27718s) [noise=true , address=77.90.31.17:4771 , piece_id=OOUSAWV2GEVAIQ4K7C3EWFX6SO3QM7O4WFY6QVXI5FB4OZ2KGYUQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 139.59.144.22:52324->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784785329 (417h33m16.277217765s) [noise=true , address=77.90.31.21:4879 , piece_id=WQDIRDNI4CFMT75KZ3TNQY5O6K5UGNTEUQUD3KM6PGLXAB2KJRVQ, seg_pos={0 0}, upload=367243]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: rpc: dial tcp 77.90.31.21:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 15 23:40:55.784818442 (417h33m16.277250887s) [seg_pos={0 0}, upload=367243]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 15 23:40:55.900501127 (417h33m16.392933562s) [upload=367243]: Done uploading segments... Sep 15 23:40:55.900528269 (417h33m16.392960711s) [upload=367243]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.919797457s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 3.231174452s FAIL (2025-09-16 00:00:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 3.231204772s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 2 out of 60 is failed (threshold=3) [..............................XX............................] duration: 224.242106ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt FAIL (2025-09-16 00:00:09 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 224.335109ms edge linksharing us-select-1

1
...
421
422
423
424
425
...
3485