Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X....X..................X...............X] duration: 30.001005236s trace: https://jaeger.dev.storj.io/trace/51e763c574611f36 FAIL (2025-10-31 21:14:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001288641s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..................X....XX...................X..............X] duration: 5.00199981s trace: https://jaeger.dev.storj.io/trace/135da41e6fd891e FAIL (2025-10-31 21:14:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002028982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.262972556s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:13:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.263001494s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...................X....XX...................X..............] duration: 1.639832677s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:13:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.63987131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X....X..................X...............] duration: 2.670429134s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:13:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.670485647s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.294337479s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:12:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.294366828s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X....X..................X..............] duration: 2.692546634s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:12:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.692584754s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....................X....XX...................X.............] duration: 1.580023498s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:11:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.580050078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.426604851s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:11:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.42663661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.330247366s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:11:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.330277347s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X....X..................X.............] duration: 2.52921217s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:11:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.529242118s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.485624342s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:11:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.48565578s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....................X....XX...................X............] duration: 1.797205774s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:10:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.797231329s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X....X..................X............] duration: 2.736122769s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:10:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.73621611s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.278240203s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:10:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 4.278264071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X....X..................X...........] duration: 2.820318404s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:10:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.820354574s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................X....XX...................X...........] duration: 2.192127944s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:10:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.192155774s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....X..................X..........] duration: 2.830938743s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:09:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.83096151s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.289798678s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:09:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.289814957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................X....XX...................X..........] duration: 1.582396514s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:09:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.582442318s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....X..................X.........] duration: 2.622558306s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:09:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.622584272s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....X..................X........] duration: 2.879206509s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:09:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.87930234s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.305410227s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:08:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.305440484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........................X....XX...................X.........] duration: 1.706111778s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:08:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.706143987s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....X..................X.......] duration: 2.760510663s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:08:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.760572988s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........................X....XX...................X........] duration: 1.64210502s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:08:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.642137417s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.386081905s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:07:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.386111382s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........................X....XX...................X.......] duration: 1.685166676s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:07:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.685188867s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................X....XX...................X......] duration: 1.635981508s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:07:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.636001737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....X..................X......] duration: 2.813707332s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:07:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.813738166s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............................X....XX...................X.....] duration: 1.565853192s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:06:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.565882247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.394858063s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:06:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.395071672s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X....X..................X.....] duration: 2.496431115s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:06:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.497321457s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.43581259s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:06:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.435844574s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............................X....XX...................X....] duration: 1.636548892s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:06:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.636600566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............................X....XX...................X...] duration: 1.653519775s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:05:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.653583142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X....X..................X....] duration: 2.519508673s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:05:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.519557032s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.375113094s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:05:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.375134766s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X....X..................X...] duration: 2.620134939s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:05:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.620166441s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.558195918s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:05:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.558219149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............................X....XX...................X..] duration: 1.617898045s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:04:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.617927877s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................................X....XX...................X.] duration: 1.754266538s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:04:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.754296744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X....X..................X..] duration: 2.628196215s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:04:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.629380125s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.882367818s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:04:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 2.882388271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X....X..................X.] duration: 9.547538893s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:03:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.547564236s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.104715323s trace: https://jaeger.dev.storj.io/trace/3190c37ec1de204b FAIL (2025-10-31 21:03:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== ece... Oct 31 21:02:59.084538100 (33h22m1.786130551s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085146938 (33h22m1.786739397s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.085627572 (33h22m1.787220026s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086232255 (33h22m1.787824711s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.086680788 (33h22m1.788273224s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087171830 (33h22m1.788764285s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.087644597 (33h22m1.789237035s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088114677 (33h22m1.789707115s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.088912483 (33h22m1.79050492s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.090696048 (33h22m1.792288503s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.092656864 (33h22m1.794249322s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Uploading piece... Oct 31 21:02:59.612239945 (33h22m2.313832412s) [upload=29363]: Next returned nil segment Oct 31 21:02:59.612259766 (33h22m2.313852242s) [upload=29363]: Waiting for error group managing segments... Oct 31 21:02:59.782145631 (33h22m2.483738081s) [noise=true , address=145.79.227.16:4916 , piece_id=GAQRLP7TNGMMA7B7KY432XEEGGGTFZ4UTEJFROSI2CW5ODPNLOIA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.812513533 (33h22m2.514105997s) [noise=true , address=145.79.227.34:4937 , piece_id=ZF3XVZ2CVP5XQBWZDNQJSBE5WU7TPTHGVHDEDCXOCOGALPZDTPNA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.814872490 (33h22m2.516464954s) [noise=true , address=145.79.227.3:4169 , piece_id=727RMVMELXFFYY5A2H4UF5WNLDDZYIGM26IPNV5FWUHTWVEBJYPQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.819283993 (33h22m2.520876447s) [noise=true , address=145.79.227.44:4196 , piece_id=MKBS52LFXXLTLYW2JLOJLE4Y5HQ4KGTUQBZVPKP765LCQ3OMPJAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.820455484 (33h22m2.522047935s) [noise=true , address=145.79.227.12:4778 , piece_id=65QYIVZ4NUYPZ72NDXXMP42BGMSRROZTGK3DFVNDMNCD5PTVEKTQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.821605719 (33h22m2.523198175s) [noise=true , address=145.79.227.29:4968 , piece_id=DFCMZNEW5DUOMF4BSA7SKO3C3MYPLOCWI4CHC2TKGJIDTGZC54NA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:02:59.825251537 (33h22m2.526843992s) [noise=true , address=145.79.227.49:4367 , piece_id=IKMNIZA2KSJMY3GRGYRQD5L64HFEGK7RZMO72SQTZWEC6CI5HD6Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:00.270618687 (33h22m2.972211139s) [noise=true , address=145.79.227.5:4246 , piece_id=PDZXF4V4RIG54NAQ4RYE7NYBIJGZCXANSNB277I7FNMEAGJMGB3Q, seg_pos={0 0}, upload=29363]: Done uploading piece. err: <nil> Oct 31 21:03:58.695390672 (33h23m1.396983132s) [noise=true , address=77.90.31.18:4813 , piece_id=FGIQ34I2YHSXRZSYY3VWRBIOJYYVQ5NUM5JQWS73VKI4IIORNGHA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:128WthA1x9wp1dV1vkpGnoveZQpU9CSgDCZfw5GMQuVpCNSVAM5, address:77.90.31.18:4813): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695409599 (33h23m1.39700206s) [noise=true , address=77.90.31.11:4577 , piece_id=EMUVRBK2V5PL74RWFMLNPBQLYRQFXOL63QJLTOVH5VCSW6KK7KRA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47832->77.90.31.11:4577: use of closed network connection Oct 31 21:03:58.695592619 (33h23m1.397185074s) [noise=true , address=77.90.31.10:4533 , piece_id=AL42ONRIYTCTVM7GJJWRA4S5JALIYKKHRMPWFC325V2BFVHFMZMA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1RDgg3m5Febnz5CQKgkHvVYKBAzJG341rrT6pMchR1XxxEb7KS, address:77.90.31.10:4533): protocol: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46474->77.90.31.10:4533: use of closed network connection Oct 31 21:03:58.695624541 (33h23m1.397216997s) [noise=true , address=45.91.116.11:4144 , piece_id=GJDNDJIZ3O2XOIJGAH75G2IBYB5N53X6FJWJGQKK5TCCG4ABN35A, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54952->45.91.116.11:4144: use of closed network connection Oct 31 21:03:58.695758218 (33h23m1.397350678s) [noise=true , address=77.90.31.17:4801 , piece_id=ZZXD6SNEHBXK3RV7OFLMUF7XVOORSEHEO3JJ7LIQZXAZ6MXPQ3VA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:127QRLNwZLcR5gnAgZrCjSUsH5T4jESmZ4kWeAnpj9YxaTSjoYe, address:77.90.31.17:4801): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.695820302 (33h23m1.397412751s) [noise=true , address=45.91.116.4:4950 , piece_id=ITWKLCDOLBF3UAG6QF4WKFJA4MF2OWYFQKEQMM2LX4UE3PNQ7XLQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1yLCMLdMxDT9HviM2QRVvP1hkektw7WmGxm8QVpp7h7ZVexkjy, address:45.91.116.4:4950): protocol: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60474->45.91.116.4:4950: use of closed network connection Oct 31 21:03:58.695922974 (33h23m1.397515426s) [noise=true , address=77.90.31.12:4613 , piece_id=3QBO4HH2MPHXF5M74RATOKF54MVRLV5X4SDP2BVN4HRA6BPLOOIQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1223QFHVZJ198KV6fEfCeyEzt7C3qP2zY84ocB71DJzY61rAMGF, address:77.90.31.12:4613): protocol: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41958->77.90.31.12:4613: use of closed network connection Oct 31 21:03:58.696059501 (33h23m1.397651954s) [noise=true , address=45.91.116.12:4246 , piece_id=KVQ32YFTT7MKB5KZRJ3ECGUCRSKYICPVYAMEZDDIHJURCVTCQMCQ, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, address:45.91.116.12:4246): protocol: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53816->45.91.116.12:4246: use of closed network connection Oct 31 21:03:58.696338013 (33h23m1.397930473s) [noise=true , address=45.91.116.10:4087 , piece_id=ZSXJ6ZW7BJRR76K6SNFJDUN37ZU7HKNYF22XPHNGQ5DQ67KCHFAA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:125T5WzC4RDcp21oJ8zBMMBq5uw2X32Wf5FszZiWUdsSR95Pd2H, address:45.91.116.10:4087): protocol: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45380->45.91.116.10:4087: use of closed network connection Oct 31 21:03:58.696475069 (33h23m1.398067525s) [noise=true , address=77.90.31.13:4647 , piece_id=AJZYQZ6ZT4IJM5QK6O3K6QAQ5VBATWU7AORQFVCDVJ6VM7I7IJKA, seg_pos={0 0}, upload=29363]: Done uploading piece. err: ecclient: upload failed (node:121XgCUbGhKnfnTDNasmpgsQzzkZAqnFie53A7pB793wKT9VCZv, address:77.90.31.13:4647): protocol: expected piece hash; context deadline exceeded; EOF Oct 31 21:03:58.696510314 (33h23m1.398117734s) [seg_pos={0 0}, upload=29363]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Oct 31 21:03:58.797254185 (33h23m1.498846641s) [upload=29363]: Done uploading segments... Oct 31 21:03:58.797316297 (33h23m1.498908754s) [upload=29363]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.104808571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................................X....XX...................X] duration: 5.000514125s trace: https://jaeger.dev.storj.io/trace/45008fef520d8189 FAIL (2025-10-31 21:03:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00053405s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X....X..................X] duration: 30.001794624s trace: https://jaeger.dev.storj.io/trace/4bac96589bc10ccc FAIL (2025-10-31 21:03:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00183058s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X....X..................] duration: 2.677048737s trace: https://jaeger.dev.storj.io/trace/783c537abed6e500 FAIL (2025-10-31 21:02:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.677076374s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.................................X....XX...................] duration: 1.643763319s trace: https://jaeger.dev.storj.io/trace/31050109a7b601f1 FAIL (2025-10-31 21:02:29 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.644344339s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)