Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X.X.......X..X.....] duration: 2.693345257s FAIL (2025-09-22 02:30:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.693372777s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..............................XXXXXXX..X.X...XXXXXXX.X.X.X.X] duration: 30.107837305s FAIL (2025-09-22 02:30:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:29:37.271405158 (564h21m57.76383759s) [upload=496489]: Starting upload Sep 22 02:29:37.271493232 (564h21m57.763925662s) [upload=496489]: Done starting upload Sep 22 02:29:37.271805078 (564h21m57.764237529s) [upload=496489]: Uploading segments... Sep 22 02:29:37.271878790 (564h21m57.764311244s) [upload=496489]: Got next segment. Inline: false Sep 22 02:29:37.271957682 (564h21m57.764390118s) [upload=496489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:29:37.271965590 (564h21m57.764398037s) [upload=496489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:29:37.386818851 (564h21m57.879251296s) [seg_pos={0 0}, upload=496489]: Begin upload segment... Sep 22 02:29:37.386955576 (564h21m57.879388022s) [noise=true , address=145.79.227.7:4295 , piece_id=LS5REBT3O4M63LSFQYVQ2FE3TTKUYSM7NEECINR73RCFXNGFZVUQ, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.387486316 (564h21m57.879918762s) [noise=true , address=45.91.116.14:4189 , piece_id=H34KQNFFVGQTNLBFJTGOGVUZFYRORQ7DCYY57JMJD3SWTXUXECLA, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.387815126 (564h21m57.880247568s) [noise=true , address=77.90.31.15:4698 , piece_id=H3RXBLRGKHMT7NBAQYMUYJZHHASOBNSOOY5BGQW2FFYVOXLPUCHA, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.388168954 (564h21m57.880601396s) [noise=true , address=77.90.31.15:4699 , piece_id=4NDMAXDQJT6ZTD77US5Y3BLAIDUZRAACPGJCNA2N5FWFYIIAWOFA, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.388486334 (564h21m57.880918765s) [noise=true , address=45.91.116.9:4263 , piece_id=6GCOHMTE56WUOR5HRV5Q5JM3NZMDT4FEW7VZ2GQBXN2DDUHC7IXQ, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.388807621 (564h21m57.881240067s) [noise=true , address=45.91.116.10:4081 , piece_id=43VH6QJL35YUW72UMAZKKH5IB3GK2ZFUXPXZN53K7AK4CGD6IDVA, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.389160482 (564h21m57.881592927s) [noise=true , address=77.90.31.10:4518 , piece_id=GOASEOEHYYYGM7OCCMGGNN5MMEQVRPAQLQPESC3WMN4J2WOFIBHQ, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.389479647 (564h21m57.88191209s) [noise=true , address=45.91.116.11:4119 , piece_id=RA4T2P6ODLVYFDEGXTEMZPJ2YDVXALYQ2QZTG3TCGY73QYLK75KA, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.389998453 (564h21m57.882430905s) [seg_pos={0 0}, upload=496489]: Done begin upload segment. Sep 22 02:29:37.390065826 (564h21m57.882498277s) [noise=true , address=77.90.31.18:4805 , piece_id=PSAQIBNKLA3SW2W7OFO3ENIHVQVCQQDOUNGQVC7RBBI6YDMAF7VQ, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.391169941 (564h21m57.883602384s) [noise=true , address=77.90.31.19:4841 , piece_id=SBRZ74DE4FGMH33CLUVREMYP4YM367JQDAOXZAWQZVXBC4DF7ZJA, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.391481122 (564h21m57.883913565s) [noise=true , address=145.79.227.6:4259 , piece_id=OF7B3I5Z2M2EJ65N23D3HTQLQOJSPEAUIURK2N2GG7D3NWBHXDHQ, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.391761223 (564h21m57.884193665s) [noise=true , address=45.91.116.11:4118 , piece_id=J6U57V5QOEHLJV6MXJL2ZFNCDUENZUZ33Q3F7LIDUENPI26RI6XA, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.392082232 (564h21m57.884514677s) [noise=true , address=77.90.31.14:4663 , piece_id=3LPOQ6BW76SFBMES4XQDBJYMLHKKDCLQYFJYYAKKKCW63HXUTLJA, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.394371527 (564h21m57.88680397s) [noise=true , address=145.79.227.17:4969 , piece_id=24THE7QIV3EPVWHPMAJ65FDZ6TBXQA7QEZAWLS6HLY5EWNW7C5SQ, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.395583061 (564h21m57.888015522s) [noise=true , address=77.90.31.12:4590 , piece_id=AMMHYB45HZVUW6NG7VYAINDC3WDM2VNXOQFO5GZ4JLN3BILCMLKQ, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:37.396039733 (564h21m57.888472178s) [noise=true , address=145.79.227.2:4114 , piece_id=PRPQPKOCIJELE46UMHC4TOQY2BRKJH6YXLUOIKT5QVMBWO7F66PQ, seg_pos={0 0}, upload=496489]: Uploading piece... Sep 22 02:29:38.001220797 (564h21m58.493653267s) [upload=496489]: Next returned nil segment Sep 22 02:29:38.001244763 (564h21m58.49367724s) [upload=496489]: Waiting for error group managing segments... Sep 22 02:29:38.132661953 (564h21m58.625094391s) [noise=true , address=45.91.116.14:4189 , piece_id=H34KQNFFVGQTNLBFJTGOGVUZFYRORQ7DCYY57JMJD3SWTXUXECLA, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.140328569 (564h21m58.632761006s) [noise=true , address=77.90.31.19:4841 , piece_id=SBRZ74DE4FGMH33CLUVREMYP4YM367JQDAOXZAWQZVXBC4DF7ZJA, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.140539669 (564h21m58.632972105s) [noise=true , address=77.90.31.10:4518 , piece_id=GOASEOEHYYYGM7OCCMGGNN5MMEQVRPAQLQPESC3WMN4J2WOFIBHQ, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.146278673 (564h21m58.638711115s) [noise=true , address=45.91.116.10:4081 , piece_id=43VH6QJL35YUW72UMAZKKH5IB3GK2ZFUXPXZN53K7AK4CGD6IDVA, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.151345704 (564h21m58.64377814s) [noise=true , address=77.90.31.15:4699 , piece_id=4NDMAXDQJT6ZTD77US5Y3BLAIDUZRAACPGJCNA2N5FWFYIIAWOFA, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.151473658 (564h21m58.643906094s) [noise=true , address=77.90.31.12:4590 , piece_id=AMMHYB45HZVUW6NG7VYAINDC3WDM2VNXOQFO5GZ4JLN3BILCMLKQ, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.152936468 (564h21m58.645368905s) [noise=true , address=45.91.116.9:4263 , piece_id=6GCOHMTE56WUOR5HRV5Q5JM3NZMDT4FEW7VZ2GQBXN2DDUHC7IXQ, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.153820707 (564h21m58.646253143s) [noise=true , address=45.91.116.11:4119 , piece_id=RA4T2P6ODLVYFDEGXTEMZPJ2YDVXALYQ2QZTG3TCGY73QYLK75KA, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.161579704 (564h21m58.654012136s) [noise=true , address=45.91.116.11:4118 , piece_id=J6U57V5QOEHLJV6MXJL2ZFNCDUENZUZ33Q3F7LIDUENPI26RI6XA, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.162338535 (564h21m58.654770969s) [noise=true , address=77.90.31.15:4698 , piece_id=H3RXBLRGKHMT7NBAQYMUYJZHHASOBNSOOY5BGQW2FFYVOXLPUCHA, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.162933682 (564h21m58.655366118s) [noise=true , address=77.90.31.14:4663 , piece_id=3LPOQ6BW76SFBMES4XQDBJYMLHKKDCLQYFJYYAKKKCW63HXUTLJA, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:29:38.172942053 (564h21m58.665374489s) [noise=true , address=77.90.31.18:4805 , piece_id=PSAQIBNKLA3SW2W7OFO3ENIHVQVCQQDOUNGQVC7RBBI6YDMAF7VQ, seg_pos={0 0}, upload=496489]: Done uploading piece. err: <nil> Sep 22 02:30:06.983585620 (564h22m27.476018061s) [noise=true , address=145.79.227.17:4969 , piece_id=24THE7QIV3EPVWHPMAJ65FDZ6TBXQA7QEZAWLS6HLY5EWNW7C5SQ, seg_pos={0 0}, upload=496489]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:30:06.983631379 (564h22m27.476063808s) [noise=true , address=145.79.227.7:4295 , piece_id=LS5REBT3O4M63LSFQYVQ2FE3TTKUYSM7NEECINR73RCFXNGFZVUQ, seg_pos={0 0}, upload=496489]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:30:06.983694401 (564h22m27.476126829s) [noise=true , address=145.79.227.6:4259 , piece_id=OF7B3I5Z2M2EJ65N23D3HTQLQOJSPEAUIURK2N2GG7D3NWBHXDHQ, seg_pos={0 0}, upload=496489]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: rpc: dial tcp 145.79.227.6:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:30:06.983767476 (564h22m27.476199921s) [noise=true , address=145.79.227.2:4114 , piece_id=PRPQPKOCIJELE46UMHC4TOQY2BRKJH6YXLUOIKT5QVMBWO7F66PQ, seg_pos={0 0}, upload=496489]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: rpc: dial tcp 145.79.227.2:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:30:06.983812333 (564h22m27.476244781s) [seg_pos={0 0}, upload=496489]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:30:07.088940026 (564h22m27.581372479s) [upload=496489]: Done uploading segments... Sep 22 02:30:07.088988117 (564h22m27.581420564s) [upload=496489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.107923543s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X.X.......X..X....] duration: 2.780742961s FAIL (2025-09-22 02:29:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.780838435s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XXXX.X....X..................] duration: 2.503869152s FAIL (2025-09-22 02:29:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.503971745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...............................XXXXXXX..X.X...XXXXXXX.X.X.X.] duration: 2.430981622s FAIL (2025-09-22 02:29:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:28:41.614134456 (564h21m2.106566907s) [upload=496477]: Starting upload Sep 22 02:28:41.614236876 (564h21m2.106669328s) [upload=496477]: Done starting upload Sep 22 02:28:41.614365924 (564h21m2.106798365s) [upload=496477]: Uploading segments... Sep 22 02:28:41.614881161 (564h21m2.107313614s) [upload=496477]: Got next segment. Inline: false Sep 22 02:28:41.614945090 (564h21m2.107377543s) [upload=496477]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:28:41.614951546 (564h21m2.107383999s) [upload=496477]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:28:41.733679537 (564h21m2.226111989s) [seg_pos={0 0}, upload=496477]: Begin upload segment... Sep 22 02:28:41.733755011 (564h21m2.226187462s) [seg_pos={0 0}, upload=496477]: Done begin upload segment. Sep 22 02:28:41.733782895 (564h21m2.226215343s) [noise=true , address=45.91.116.14:4189 , piece_id=IGJ6WCC6M4CZRPT6W4KQKES5DYUHPC4LNDZNROZO5OV5SS3LUE7A, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.733859658 (564h21m2.226292121s) [noise=true , address=77.90.31.14:4663 , piece_id=ISNKDC23NJHL3GHF5Q5P56ZBDPEY374KSVZ562CDNYWQ3YOHZALA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.734478161 (564h21m2.226910626s) [noise=true , address=145.79.227.2:4114 , piece_id=O2APXZNOY2FGTZD6P3JMFJOONJACFG6MDP6ICLO7RRQZO45AGY4A, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.734968727 (564h21m2.227401182s) [noise=true , address=77.90.31.16:4734 , piece_id=YKMU5FLSZWJI4FKCKSD33SOCZRG3XBOQHAWOVH6J7BXYJTT6HZPQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.735408147 (564h21m2.227840599s) [noise=true , address=145.79.227.17:4972 , piece_id=TEGQPU7E2VHAC5S5Q62P5G7VT73ARGPDI3M7IZ6LGSEPVMD2QVDA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.735830976 (564h21m2.228263427s) [noise=true , address=145.79.227.13:4804 , piece_id=WQOFYFBUXWXJWUPFEGDUTK43QGMFP32O5SETRJVWWJQRZWIQCPPQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.737659559 (564h21m2.230092028s) [noise=true , address=45.91.116.9:4262 , piece_id=IMLKCRSPFBMIHGJI4QYN5HBODEGE2EWUM3XZD4GSHN523IZZHAKA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.738037084 (564h21m2.230469533s) [noise=true , address=45.91.116.9:4263 , piece_id=JPPEAVLPOUNR6ZGUYBXFXZGPSDPLBLZW5OFI24G6KEHAXNFVITPQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.738487674 (564h21m2.230920123s) [noise=true , address=45.91.116.12:4227 , piece_id=7UDG5JMP54X7YNTXLMU5LN4UTPW4R7NFQGGZSI2N4QVOADXLXF2A, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.738785392 (564h21m2.231217839s) [noise=true , address=77.90.31.17:4772 , piece_id=WPGP3XPAAMJQBVMZXXVQDZJQFZZIX7BZWFSBQVNY2VBRHDQTW35A, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.739073738 (564h21m2.231506181s) [noise=true , address=45.91.116.12:4226 , piece_id=ZMRQDBLOOFTQGZ2C4Z6YS4QF6J4SKPKX5MPV2OBZBFDNZBF36EMA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.739376574 (564h21m2.231809027s) [noise=true , address=45.91.116.6:4975 , piece_id=VU22FJG2ROTBTCHXQUGHGFUNIAJA3SO7Q2NBMWKJ5O3X5LCWSLCA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.739696713 (564h21m2.232129159s) [noise=true , address=45.91.116.8:4045 , piece_id=PUCR6JEIAB6L5Y53FH44NDSPVCMVXLU2FMCTOLABM7R5PBR7HOMQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.739972337 (564h21m2.232404782s) [noise=true , address=45.91.116.10:4082 , piece_id=I7DXHK46SGAAAWLBZPDSFP2F6AZUG3CDEKKQ5FHUGXBQHUKYI6RQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.740211044 (564h21m2.232643485s) [noise=true , address=145.79.227.14:4839 , piece_id=BSPGXRH2OCYGR4NYFJ5GGCDXD72ARAPMIBRIX7RZXGIPOEP6OPYQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.740519491 (564h21m2.232951944s) [noise=true , address=77.90.31.13:4626 , piece_id=N72VFAOSMUV5TFUKCNTZEF4EX4QBOQYJC2GAGW7AP52KNSX4NWQQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:42.341528275 (564h21m2.833960703s) [upload=496477]: Next returned nil segment Sep 22 02:28:42.341574767 (564h21m2.834007215s) [upload=496477]: Waiting for error group managing segments... Sep 22 02:28:42.482175529 (564h21m2.974607979s) [noise=true , address=45.91.116.6:4975 , piece_id=VU22FJG2ROTBTCHXQUGHGFUNIAJA3SO7Q2NBMWKJ5O3X5LCWSLCA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.483282017 (564h21m2.975714453s) [noise=true , address=45.91.116.12:4226 , piece_id=ZMRQDBLOOFTQGZ2C4Z6YS4QF6J4SKPKX5MPV2OBZBFDNZBF36EMA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.485016431 (564h21m2.977448866s) [noise=true , address=45.91.116.14:4189 , piece_id=IGJ6WCC6M4CZRPT6W4KQKES5DYUHPC4LNDZNROZO5OV5SS3LUE7A, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.487496471 (564h21m2.979928922s) [noise=true , address=77.90.31.17:4772 , piece_id=WPGP3XPAAMJQBVMZXXVQDZJQFZZIX7BZWFSBQVNY2VBRHDQTW35A, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.487979633 (564h21m2.980412069s) [noise=true , address=45.91.116.8:4045 , piece_id=PUCR6JEIAB6L5Y53FH44NDSPVCMVXLU2FMCTOLABM7R5PBR7HOMQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.488473126 (564h21m2.980905576s) [noise=true , address=45.91.116.12:4227 , piece_id=7UDG5JMP54X7YNTXLMU5LN4UTPW4R7NFQGGZSI2N4QVOADXLXF2A, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.493226542 (564h21m2.98565897s) [noise=true , address=45.91.116.10:4082 , piece_id=I7DXHK46SGAAAWLBZPDSFP2F6AZUG3CDEKKQ5FHUGXBQHUKYI6RQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.494666828 (564h21m2.987099257s) [noise=true , address=77.90.31.16:4734 , piece_id=YKMU5FLSZWJI4FKCKSD33SOCZRG3XBOQHAWOVH6J7BXYJTT6HZPQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.498177759 (564h21m2.990610208s) [noise=true , address=45.91.116.9:4263 , piece_id=JPPEAVLPOUNR6ZGUYBXFXZGPSDPLBLZW5OFI24G6KEHAXNFVITPQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.510583836 (564h21m3.003016276s) [noise=true , address=45.91.116.9:4262 , piece_id=IMLKCRSPFBMIHGJI4QYN5HBODEGE2EWUM3XZD4GSHN523IZZHAKA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.510688509 (564h21m3.003120953s) [noise=true , address=77.90.31.13:4626 , piece_id=N72VFAOSMUV5TFUKCNTZEF4EX4QBOQYJC2GAGW7AP52KNSX4NWQQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.512246906 (564h21m3.004679334s) [noise=true , address=77.90.31.14:4663 , piece_id=ISNKDC23NJHL3GHF5Q5P56ZBDPEY374KSVZ562CDNYWQ3YOHZALA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:29:11.312839361 (564h21m31.805271808s) [noise=true , address=145.79.227.14:4839 , piece_id=BSPGXRH2OCYGR4NYFJ5GGCDXD72ARAPMIBRIX7RZXGIPOEP6OPYQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:29:11.312954770 (564h21m31.805387219s) [noise=true , address=145.79.227.13:4804 , piece_id=WQOFYFBUXWXJWUPFEGDUTK43QGMFP32O5SETRJVWWJQRZWIQCPPQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: rpc: dial tcp 145.79.227.13:4804: i/o timeout; conn closed: op Sep 22 02:29:11.312999086 (564h21m31.80543153s) [noise=true , address=145.79.227.2:4114 , piece_id=O2APXZNOY2FGTZD6P3JMFJOONJACFG6MDP6ICLO7RRQZO45AGY4A, seg_pos={0 0}, upload=496477]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: rpc: dial tcp 145.79.227.2:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:29:11.313028469 (564h21m31.805460912s) [noise=true , address=145.79.227.17:4972 , piece_id=TEGQPU7E2VHAC5S5Q62P5G7VT73ARGPDI3M7IZ6LGSEPVMD2QVDA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:29:11.313051406 (564h21m31.805483855s) [seg_pos={0 0}, upload=496477]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:29:11.421489347 (564h21m31.913921784s) [upload=496477]: Done uploading segments... Sep 22 02:29:11.421519831 (564h21m31.913952273s) [upload=496477]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.431006848s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [................................XXXXXXX..X.X...XXXXXXX.X.X.X] duration: 30.110472941s FAIL (2025-09-22 02:29:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:28:41.614134456 (564h21m2.106566907s) [upload=496477]: Starting upload Sep 22 02:28:41.614236876 (564h21m2.106669328s) [upload=496477]: Done starting upload Sep 22 02:28:41.614365924 (564h21m2.106798365s) [upload=496477]: Uploading segments... Sep 22 02:28:41.614881161 (564h21m2.107313614s) [upload=496477]: Got next segment. Inline: false Sep 22 02:28:41.614945090 (564h21m2.107377543s) [upload=496477]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:28:41.614951546 (564h21m2.107383999s) [upload=496477]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:28:41.733679537 (564h21m2.226111989s) [seg_pos={0 0}, upload=496477]: Begin upload segment... Sep 22 02:28:41.733755011 (564h21m2.226187462s) [seg_pos={0 0}, upload=496477]: Done begin upload segment. Sep 22 02:28:41.733782895 (564h21m2.226215343s) [noise=true , address=45.91.116.14:4189 , piece_id=IGJ6WCC6M4CZRPT6W4KQKES5DYUHPC4LNDZNROZO5OV5SS3LUE7A, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.733859658 (564h21m2.226292121s) [noise=true , address=77.90.31.14:4663 , piece_id=ISNKDC23NJHL3GHF5Q5P56ZBDPEY374KSVZ562CDNYWQ3YOHZALA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.734478161 (564h21m2.226910626s) [noise=true , address=145.79.227.2:4114 , piece_id=O2APXZNOY2FGTZD6P3JMFJOONJACFG6MDP6ICLO7RRQZO45AGY4A, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.734968727 (564h21m2.227401182s) [noise=true , address=77.90.31.16:4734 , piece_id=YKMU5FLSZWJI4FKCKSD33SOCZRG3XBOQHAWOVH6J7BXYJTT6HZPQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.735408147 (564h21m2.227840599s) [noise=true , address=145.79.227.17:4972 , piece_id=TEGQPU7E2VHAC5S5Q62P5G7VT73ARGPDI3M7IZ6LGSEPVMD2QVDA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.735830976 (564h21m2.228263427s) [noise=true , address=145.79.227.13:4804 , piece_id=WQOFYFBUXWXJWUPFEGDUTK43QGMFP32O5SETRJVWWJQRZWIQCPPQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.737659559 (564h21m2.230092028s) [noise=true , address=45.91.116.9:4262 , piece_id=IMLKCRSPFBMIHGJI4QYN5HBODEGE2EWUM3XZD4GSHN523IZZHAKA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.738037084 (564h21m2.230469533s) [noise=true , address=45.91.116.9:4263 , piece_id=JPPEAVLPOUNR6ZGUYBXFXZGPSDPLBLZW5OFI24G6KEHAXNFVITPQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.738487674 (564h21m2.230920123s) [noise=true , address=45.91.116.12:4227 , piece_id=7UDG5JMP54X7YNTXLMU5LN4UTPW4R7NFQGGZSI2N4QVOADXLXF2A, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.738785392 (564h21m2.231217839s) [noise=true , address=77.90.31.17:4772 , piece_id=WPGP3XPAAMJQBVMZXXVQDZJQFZZIX7BZWFSBQVNY2VBRHDQTW35A, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.739073738 (564h21m2.231506181s) [noise=true , address=45.91.116.12:4226 , piece_id=ZMRQDBLOOFTQGZ2C4Z6YS4QF6J4SKPKX5MPV2OBZBFDNZBF36EMA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.739376574 (564h21m2.231809027s) [noise=true , address=45.91.116.6:4975 , piece_id=VU22FJG2ROTBTCHXQUGHGFUNIAJA3SO7Q2NBMWKJ5O3X5LCWSLCA, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.739696713 (564h21m2.232129159s) [noise=true , address=45.91.116.8:4045 , piece_id=PUCR6JEIAB6L5Y53FH44NDSPVCMVXLU2FMCTOLABM7R5PBR7HOMQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.739972337 (564h21m2.232404782s) [noise=true , address=45.91.116.10:4082 , piece_id=I7DXHK46SGAAAWLBZPDSFP2F6AZUG3CDEKKQ5FHUGXBQHUKYI6RQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.740211044 (564h21m2.232643485s) [noise=true , address=145.79.227.14:4839 , piece_id=BSPGXRH2OCYGR4NYFJ5GGCDXD72ARAPMIBRIX7RZXGIPOEP6OPYQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:41.740519491 (564h21m2.232951944s) [noise=true , address=77.90.31.13:4626 , piece_id=N72VFAOSMUV5TFUKCNTZEF4EX4QBOQYJC2GAGW7AP52KNSX4NWQQ, seg_pos={0 0}, upload=496477]: Uploading piece... Sep 22 02:28:42.341528275 (564h21m2.833960703s) [upload=496477]: Next returned nil segment Sep 22 02:28:42.341574767 (564h21m2.834007215s) [upload=496477]: Waiting for error group managing segments... Sep 22 02:28:42.482175529 (564h21m2.974607979s) [noise=true , address=45.91.116.6:4975 , piece_id=VU22FJG2ROTBTCHXQUGHGFUNIAJA3SO7Q2NBMWKJ5O3X5LCWSLCA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.483282017 (564h21m2.975714453s) [noise=true , address=45.91.116.12:4226 , piece_id=ZMRQDBLOOFTQGZ2C4Z6YS4QF6J4SKPKX5MPV2OBZBFDNZBF36EMA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.485016431 (564h21m2.977448866s) [noise=true , address=45.91.116.14:4189 , piece_id=IGJ6WCC6M4CZRPT6W4KQKES5DYUHPC4LNDZNROZO5OV5SS3LUE7A, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.487496471 (564h21m2.979928922s) [noise=true , address=77.90.31.17:4772 , piece_id=WPGP3XPAAMJQBVMZXXVQDZJQFZZIX7BZWFSBQVNY2VBRHDQTW35A, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.487979633 (564h21m2.980412069s) [noise=true , address=45.91.116.8:4045 , piece_id=PUCR6JEIAB6L5Y53FH44NDSPVCMVXLU2FMCTOLABM7R5PBR7HOMQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.488473126 (564h21m2.980905576s) [noise=true , address=45.91.116.12:4227 , piece_id=7UDG5JMP54X7YNTXLMU5LN4UTPW4R7NFQGGZSI2N4QVOADXLXF2A, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.493226542 (564h21m2.98565897s) [noise=true , address=45.91.116.10:4082 , piece_id=I7DXHK46SGAAAWLBZPDSFP2F6AZUG3CDEKKQ5FHUGXBQHUKYI6RQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.494666828 (564h21m2.987099257s) [noise=true , address=77.90.31.16:4734 , piece_id=YKMU5FLSZWJI4FKCKSD33SOCZRG3XBOQHAWOVH6J7BXYJTT6HZPQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.498177759 (564h21m2.990610208s) [noise=true , address=45.91.116.9:4263 , piece_id=JPPEAVLPOUNR6ZGUYBXFXZGPSDPLBLZW5OFI24G6KEHAXNFVITPQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.510583836 (564h21m3.003016276s) [noise=true , address=45.91.116.9:4262 , piece_id=IMLKCRSPFBMIHGJI4QYN5HBODEGE2EWUM3XZD4GSHN523IZZHAKA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.510688509 (564h21m3.003120953s) [noise=true , address=77.90.31.13:4626 , piece_id=N72VFAOSMUV5TFUKCNTZEF4EX4QBOQYJC2GAGW7AP52KNSX4NWQQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:28:42.512246906 (564h21m3.004679334s) [noise=true , address=77.90.31.14:4663 , piece_id=ISNKDC23NJHL3GHF5Q5P56ZBDPEY374KSVZ562CDNYWQ3YOHZALA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: <nil> Sep 22 02:29:11.312839361 (564h21m31.805271808s) [noise=true , address=145.79.227.14:4839 , piece_id=BSPGXRH2OCYGR4NYFJ5GGCDXD72ARAPMIBRIX7RZXGIPOEP6OPYQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:29:11.312954770 (564h21m31.805387219s) [noise=true , address=145.79.227.13:4804 , piece_id=WQOFYFBUXWXJWUPFEGDUTK43QGMFP32O5SETRJVWWJQRZWIQCPPQ, seg_pos={0 0}, upload=496477]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: rpc: dial tcp 145.79.227.13:4804: i/o timeout; conn closed: op Sep 22 02:29:11.312999086 (564h21m31.80543153s) [noise=true , address=145.79.227.2:4114 , piece_id=O2APXZNOY2FGTZD6P3JMFJOONJACFG6MDP6ICLO7RRQZO45AGY4A, seg_pos={0 0}, upload=496477]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: rpc: dial tcp 145.79.227.2:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:29:11.313028469 (564h21m31.805460912s) [noise=true , address=145.79.227.17:4972 , piece_id=TEGQPU7E2VHAC5S5Q62P5G7VT73ARGPDI3M7IZ6LGSEPVMD2QVDA, seg_pos={0 0}, upload=496477]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:29:11.313051406 (564h21m31.805483855s) [seg_pos={0 0}, upload=496477]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:29:11.421489347 (564h21m31.913921784s) [upload=496477]: Done uploading segments... Sep 22 02:29:11.421519831 (564h21m31.913952273s) [upload=496477]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.1105272s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X.X.......X..X...] duration: 2.664478745s FAIL (2025-09-22 02:28:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.664580138s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XXXX.X....X.................] duration: 2.422315001s FAIL (2025-09-22 02:28:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.422345802s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.................................XXXXXXX..X.X...XXXXXXX.X.X.] duration: 2.435789072s FAIL (2025-09-22 02:28:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 02:27:04.747470664 (564h19m25.239903112s) [upload=496451]: Starting upload Sep 22 02:27:04.747634369 (564h19m25.240066814s) [upload=496451]: Done starting upload Sep 22 02:27:04.747719162 (564h19m25.24015159s) [upload=496451]: Uploading segments... Sep 22 02:27:04.748050247 (564h19m25.240482702s) [upload=496451]: Got next segment. Inline: false Sep 22 02:27:04.750464006 (564h19m25.242896475s) [upload=496451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:27:04.750481503 (564h19m25.242913956s) [upload=496451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:27:04.879850980 (564h19m25.372283422s) [seg_pos={0 0}, upload=496451]: Begin upload segment... Sep 22 02:27:04.879922272 (564h19m25.372354713s) [seg_pos={0 0}, upload=496451]: Done begin upload segment. Sep 22 02:27:04.879948446 (564h19m25.372380892s) [noise=true , address=45.91.116.6:4974 , piece_id=DPD63LLS6CYW2OMEYSISF6FS3JLGCZEM4IBOJIVJYV5ZBCZ2SBRA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880165077 (564h19m25.37259751s) [noise=true , address=77.90.31.17:4772 , piece_id=XJO4LYSGAM3F2SFUJR4FAU57YAZ2QPQQJQW42EDHQ2GND24IU7SQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880193776 (564h19m25.372626212s) [noise=true , address=45.91.116.8:4045 , piece_id=LA7UMWZ3KO4XIDKKMOVTFBJBRVZWSGNNSV6NMFVYKM7DIJ3SQ5TQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880220756 (564h19m25.372653185s) [noise=true , address=45.91.116.4:4938 , piece_id=IB5VSYEC2MSCEABMOGOAKVICFX2H5DIXLSFQNB3WQDCYQ2FCD5ZA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880246833 (564h19m25.372679263s) [noise=true , address=77.90.31.15:4698 , piece_id=RGQPMH62W7ZFHHDRH3ZG5XGNZQPGONWNPSIMBJOBWE3WGUZS5WFA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880275411 (564h19m25.372707855s) [noise=true , address=145.79.227.13:4801 , piece_id=R2K7AK5MAMIHM7NUOSPFGDINAJSQZE7S2CPY7Y223OW7IB7UQ6AQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880313688 (564h19m25.372746129s) [noise=true , address=45.91.116.11:4118 , piece_id=Z3E4RG4SII3SEL6JMWFDRPDPZFQB2EB2YYZ7B5FJBNHVMXPVBTRQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880347494 (564h19m25.372779928s) [noise=true , address=145.79.227.7:4294 , piece_id=7TEANGKDLASADDLR6HNY5CQWRBOKFARLWUFPYX3ZGUONBPP2RZGA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880376519 (564h19m25.372808948s) [noise=true , address=77.90.31.18:4806 , piece_id=VNMSC75YZBUKDGVRSHRL43VJME3HKCRTTIMMKZHZIYELTXBB3NXQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880406244 (564h19m25.372838694s) [noise=true , address=145.79.227.16:4912 , piece_id=E5FVQYVCL2RB2SMIQSBGDDC25Y6ONZJU6WQVHX3ESQY5NVIKQ2KQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880439102 (564h19m25.37287154s) [noise=true , address=145.79.227.16:4909 , piece_id=QJT6BY3DDE5C2SZJA2LHGBCQ5FOFREEUSMFKWUG2OZD6UYJ76ZCA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880466231 (564h19m25.372898666s) [noise=true , address=145.79.227.19:4077 , piece_id=NZ2PKBWQZMI5W7MDTCWGF3ZXI6EM4AFAA5TXBXIWEGF3ACBJR7IQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880495436 (564h19m25.372927877s) [noise=true , address=45.91.116.6:4973 , piece_id=JPJU5AOFCQ4KJNRGAAM36F32OAHXA3FGGUBHRX64UWEIOWFQYGBA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880520018 (564h19m25.372952454s) [noise=true , address=45.91.116.14:4189 , piece_id=TFKZXUC3ZCCME4UHZLG43W6YOTR7MHPWOEFW5BUXJX5FOIWAZX4Q, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880545760 (564h19m25.372978197s) [noise=true , address=145.79.227.4:4186 , piece_id=AP65KGMY3FQOJHNYCROLWAYUZSEHOFCFBLRJCXPYVS6HKO2KWK5A, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880566578 (564h19m25.372999012s) [noise=true , address=145.79.227.14:4838 , piece_id=NFZLFYEZQLLW4FXMPYKMZP32BSCAHAITFYNQIFSOEHEQ56N7SBCA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:05.498274434 (564h19m25.990706871s) [upload=496451]: Next returned nil segment Sep 22 02:27:05.498291728 (564h19m25.990724176s) [upload=496451]: Waiting for error group managing segments... Sep 22 02:27:05.634226354 (564h19m26.126658797s) [noise=true , address=45.91.116.11:4118 , piece_id=Z3E4RG4SII3SEL6JMWFDRPDPZFQB2EB2YYZ7B5FJBNHVMXPVBTRQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.635294941 (564h19m26.127727385s) [noise=true , address=45.91.116.6:4973 , piece_id=JPJU5AOFCQ4KJNRGAAM36F32OAHXA3FGGUBHRX64UWEIOWFQYGBA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.639009202 (564h19m26.13144166s) [noise=true , address=77.90.31.15:4698 , piece_id=RGQPMH62W7ZFHHDRH3ZG5XGNZQPGONWNPSIMBJOBWE3WGUZS5WFA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.644205993 (564h19m26.136638422s) [noise=true , address=45.91.116.4:4938 , piece_id=IB5VSYEC2MSCEABMOGOAKVICFX2H5DIXLSFQNB3WQDCYQ2FCD5ZA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.644278171 (564h19m26.1367106s) [noise=true , address=77.90.31.17:4772 , piece_id=XJO4LYSGAM3F2SFUJR4FAU57YAZ2QPQQJQW42EDHQ2GND24IU7SQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.644628380 (564h19m26.137060816s) [noise=true , address=45.91.116.8:4045 , piece_id=LA7UMWZ3KO4XIDKKMOVTFBJBRVZWSGNNSV6NMFVYKM7DIJ3SQ5TQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.646718493 (564h19m26.139150924s) [noise=true , address=45.91.116.6:4974 , piece_id=DPD63LLS6CYW2OMEYSISF6FS3JLGCZEM4IBOJIVJYV5ZBCZ2SBRA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.647941192 (564h19m26.140373625s) [noise=true , address=77.90.31.18:4806 , piece_id=VNMSC75YZBUKDGVRSHRL43VJME3HKCRTTIMMKZHZIYELTXBB3NXQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.648757254 (564h19m26.14118969s) [noise=true , address=45.91.116.14:4189 , piece_id=TFKZXUC3ZCCME4UHZLG43W6YOTR7MHPWOEFW5BUXJX5FOIWAZX4Q, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:34.445230461 (564h19m54.937662904s) [noise=true , address=145.79.227.16:4909 , piece_id=QJT6BY3DDE5C2SZJA2LHGBCQ5FOFREEUSMFKWUG2OZD6UYJ76ZCA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:27:34.445570473 (564h19m54.938002919s) [noise=true , address=145.79.227.7:4294 , piece_id=7TEANGKDLASADDLR6HNY5CQWRBOKFARLWUFPYX3ZGUONBPP2RZGA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445588809 (564h19m54.938021252s) [noise=true , address=145.79.227.13:4801 , piece_id=R2K7AK5MAMIHM7NUOSPFGDINAJSQZE7S2CPY7Y223OW7IB7UQ6AQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:27:34.445803639 (564h19m54.938236086s) [noise=true , address=145.79.227.4:4186 , piece_id=AP65KGMY3FQOJHNYCROLWAYUZSEHOFCFBLRJCXPYVS6HKO2KWK5A, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445818200 (564h19m54.938250632s) [noise=true , address=145.79.227.14:4838 , piece_id=NFZLFYEZQLLW4FXMPYKMZP32BSCAHAITFYNQIFSOEHEQ56N7SBCA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445888658 (564h19m54.938321092s) [noise=true , address=145.79.227.16:4912 , piece_id=E5FVQYVCL2RB2SMIQSBGDDC25Y6ONZJU6WQVHX3ESQY5NVIKQ2KQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445946843 (564h19m54.938379289s) [noise=true , address=145.79.227.19:4077 , piece_id=NZ2PKBWQZMI5W7MDTCWGF3ZXI6EM4AFAA5TXBXIWEGF3ACBJR7IQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445989305 (564h19m54.938421757s) [seg_pos={0 0}, upload=496451]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 02:27:34.568362870 (564h19m55.060795305s) [upload=496451]: Done uploading segments... Sep 22 02:27:34.568394693 (564h19m55.060827128s) [upload=496451]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.435812121s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X.X.......X..X..] duration: 2.924041672s FAIL (2025-09-22 02:27:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.924063154s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..................................XXXXXXX..X.X...XXXXXXX.X.X] duration: 30.125407372s FAIL (2025-09-22 02:27:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 02:27:04.747470664 (564h19m25.239903112s) [upload=496451]: Starting upload Sep 22 02:27:04.747634369 (564h19m25.240066814s) [upload=496451]: Done starting upload Sep 22 02:27:04.747719162 (564h19m25.24015159s) [upload=496451]: Uploading segments... Sep 22 02:27:04.748050247 (564h19m25.240482702s) [upload=496451]: Got next segment. Inline: false Sep 22 02:27:04.750464006 (564h19m25.242896475s) [upload=496451]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:27:04.750481503 (564h19m25.242913956s) [upload=496451]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:27:04.879850980 (564h19m25.372283422s) [seg_pos={0 0}, upload=496451]: Begin upload segment... Sep 22 02:27:04.879922272 (564h19m25.372354713s) [seg_pos={0 0}, upload=496451]: Done begin upload segment. Sep 22 02:27:04.879948446 (564h19m25.372380892s) [noise=true , address=45.91.116.6:4974 , piece_id=DPD63LLS6CYW2OMEYSISF6FS3JLGCZEM4IBOJIVJYV5ZBCZ2SBRA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880165077 (564h19m25.37259751s) [noise=true , address=77.90.31.17:4772 , piece_id=XJO4LYSGAM3F2SFUJR4FAU57YAZ2QPQQJQW42EDHQ2GND24IU7SQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880193776 (564h19m25.372626212s) [noise=true , address=45.91.116.8:4045 , piece_id=LA7UMWZ3KO4XIDKKMOVTFBJBRVZWSGNNSV6NMFVYKM7DIJ3SQ5TQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880220756 (564h19m25.372653185s) [noise=true , address=45.91.116.4:4938 , piece_id=IB5VSYEC2MSCEABMOGOAKVICFX2H5DIXLSFQNB3WQDCYQ2FCD5ZA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880246833 (564h19m25.372679263s) [noise=true , address=77.90.31.15:4698 , piece_id=RGQPMH62W7ZFHHDRH3ZG5XGNZQPGONWNPSIMBJOBWE3WGUZS5WFA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880275411 (564h19m25.372707855s) [noise=true , address=145.79.227.13:4801 , piece_id=R2K7AK5MAMIHM7NUOSPFGDINAJSQZE7S2CPY7Y223OW7IB7UQ6AQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880313688 (564h19m25.372746129s) [noise=true , address=45.91.116.11:4118 , piece_id=Z3E4RG4SII3SEL6JMWFDRPDPZFQB2EB2YYZ7B5FJBNHVMXPVBTRQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880347494 (564h19m25.372779928s) [noise=true , address=145.79.227.7:4294 , piece_id=7TEANGKDLASADDLR6HNY5CQWRBOKFARLWUFPYX3ZGUONBPP2RZGA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880376519 (564h19m25.372808948s) [noise=true , address=77.90.31.18:4806 , piece_id=VNMSC75YZBUKDGVRSHRL43VJME3HKCRTTIMMKZHZIYELTXBB3NXQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880406244 (564h19m25.372838694s) [noise=true , address=145.79.227.16:4912 , piece_id=E5FVQYVCL2RB2SMIQSBGDDC25Y6ONZJU6WQVHX3ESQY5NVIKQ2KQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880439102 (564h19m25.37287154s) [noise=true , address=145.79.227.16:4909 , piece_id=QJT6BY3DDE5C2SZJA2LHGBCQ5FOFREEUSMFKWUG2OZD6UYJ76ZCA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880466231 (564h19m25.372898666s) [noise=true , address=145.79.227.19:4077 , piece_id=NZ2PKBWQZMI5W7MDTCWGF3ZXI6EM4AFAA5TXBXIWEGF3ACBJR7IQ, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880495436 (564h19m25.372927877s) [noise=true , address=45.91.116.6:4973 , piece_id=JPJU5AOFCQ4KJNRGAAM36F32OAHXA3FGGUBHRX64UWEIOWFQYGBA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880520018 (564h19m25.372952454s) [noise=true , address=45.91.116.14:4189 , piece_id=TFKZXUC3ZCCME4UHZLG43W6YOTR7MHPWOEFW5BUXJX5FOIWAZX4Q, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880545760 (564h19m25.372978197s) [noise=true , address=145.79.227.4:4186 , piece_id=AP65KGMY3FQOJHNYCROLWAYUZSEHOFCFBLRJCXPYVS6HKO2KWK5A, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:04.880566578 (564h19m25.372999012s) [noise=true , address=145.79.227.14:4838 , piece_id=NFZLFYEZQLLW4FXMPYKMZP32BSCAHAITFYNQIFSOEHEQ56N7SBCA, seg_pos={0 0}, upload=496451]: Uploading piece... Sep 22 02:27:05.498274434 (564h19m25.990706871s) [upload=496451]: Next returned nil segment Sep 22 02:27:05.498291728 (564h19m25.990724176s) [upload=496451]: Waiting for error group managing segments... Sep 22 02:27:05.634226354 (564h19m26.126658797s) [noise=true , address=45.91.116.11:4118 , piece_id=Z3E4RG4SII3SEL6JMWFDRPDPZFQB2EB2YYZ7B5FJBNHVMXPVBTRQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.635294941 (564h19m26.127727385s) [noise=true , address=45.91.116.6:4973 , piece_id=JPJU5AOFCQ4KJNRGAAM36F32OAHXA3FGGUBHRX64UWEIOWFQYGBA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.639009202 (564h19m26.13144166s) [noise=true , address=77.90.31.15:4698 , piece_id=RGQPMH62W7ZFHHDRH3ZG5XGNZQPGONWNPSIMBJOBWE3WGUZS5WFA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.644205993 (564h19m26.136638422s) [noise=true , address=45.91.116.4:4938 , piece_id=IB5VSYEC2MSCEABMOGOAKVICFX2H5DIXLSFQNB3WQDCYQ2FCD5ZA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.644278171 (564h19m26.1367106s) [noise=true , address=77.90.31.17:4772 , piece_id=XJO4LYSGAM3F2SFUJR4FAU57YAZ2QPQQJQW42EDHQ2GND24IU7SQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.644628380 (564h19m26.137060816s) [noise=true , address=45.91.116.8:4045 , piece_id=LA7UMWZ3KO4XIDKKMOVTFBJBRVZWSGNNSV6NMFVYKM7DIJ3SQ5TQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.646718493 (564h19m26.139150924s) [noise=true , address=45.91.116.6:4974 , piece_id=DPD63LLS6CYW2OMEYSISF6FS3JLGCZEM4IBOJIVJYV5ZBCZ2SBRA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.647941192 (564h19m26.140373625s) [noise=true , address=77.90.31.18:4806 , piece_id=VNMSC75YZBUKDGVRSHRL43VJME3HKCRTTIMMKZHZIYELTXBB3NXQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:05.648757254 (564h19m26.14118969s) [noise=true , address=45.91.116.14:4189 , piece_id=TFKZXUC3ZCCME4UHZLG43W6YOTR7MHPWOEFW5BUXJX5FOIWAZX4Q, seg_pos={0 0}, upload=496451]: Done uploading piece. err: <nil> Sep 22 02:27:34.445230461 (564h19m54.937662904s) [noise=true , address=145.79.227.16:4909 , piece_id=QJT6BY3DDE5C2SZJA2LHGBCQ5FOFREEUSMFKWUG2OZD6UYJ76ZCA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:27:34.445570473 (564h19m54.938002919s) [noise=true , address=145.79.227.7:4294 , piece_id=7TEANGKDLASADDLR6HNY5CQWRBOKFARLWUFPYX3ZGUONBPP2RZGA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445588809 (564h19m54.938021252s) [noise=true , address=145.79.227.13:4801 , piece_id=R2K7AK5MAMIHM7NUOSPFGDINAJSQZE7S2CPY7Y223OW7IB7UQ6AQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:27:34.445803639 (564h19m54.938236086s) [noise=true , address=145.79.227.4:4186 , piece_id=AP65KGMY3FQOJHNYCROLWAYUZSEHOFCFBLRJCXPYVS6HKO2KWK5A, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445818200 (564h19m54.938250632s) [noise=true , address=145.79.227.14:4838 , piece_id=NFZLFYEZQLLW4FXMPYKMZP32BSCAHAITFYNQIFSOEHEQ56N7SBCA, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:145.79.227.14:4838): protocol: rpc: dial tcp 145.79.227.14:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445888658 (564h19m54.938321092s) [noise=true , address=145.79.227.16:4912 , piece_id=E5FVQYVCL2RB2SMIQSBGDDC25Y6ONZJU6WQVHX3ESQY5NVIKQ2KQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445946843 (564h19m54.938379289s) [noise=true , address=145.79.227.19:4077 , piece_id=NZ2PKBWQZMI5W7MDTCWGF3ZXI6EM4AFAA5TXBXIWEGF3ACBJR7IQ, seg_pos={0 0}, upload=496451]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:27:34.445989305 (564h19m54.938421757s) [seg_pos={0 0}, upload=496451]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 02:27:34.568362870 (564h19m55.060795305s) [upload=496451]: Done uploading segments... Sep 22 02:27:34.568394693 (564h19m55.060827128s) [upload=496451]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.125444482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................XXXX.X....X................] duration: 2.539426731s FAIL (2025-09-22 02:27:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.539519053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................XXXX.X....X...............] duration: 2.570789454s FAIL (2025-09-22 02:27:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.570827069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X.X.......X..X.] duration: 2.671017252s FAIL (2025-09-22 02:26:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.671051272s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...................................XXXXXXX..X.X...XXXXXXX.X.] duration: 2.481693669s FAIL (2025-09-22 02:26:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 02:25:55.020025511 (564h18m15.512457939s) [upload=496433]: Starting upload Sep 22 02:25:55.020111359 (564h18m15.512543804s) [upload=496433]: Done starting upload Sep 22 02:25:55.020234446 (564h18m15.512666881s) [upload=496433]: Uploading segments... Sep 22 02:25:55.020416850 (564h18m15.512849284s) [upload=496433]: Got next segment. Inline: false Sep 22 02:25:55.020465495 (564h18m15.512897949s) [upload=496433]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:25:55.020474934 (564h18m15.51290738s) [upload=496433]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:25:55.133086277 (564h18m15.625518729s) [seg_pos={0 0}, upload=496433]: Begin upload segment... Sep 22 02:25:55.133165552 (564h18m15.625597996s) [seg_pos={0 0}, upload=496433]: Done begin upload segment. Sep 22 02:25:55.133210341 (564h18m15.625642792s) [noise=true , address=145.79.227.7:4295 , piece_id=O42XY6PSEQS2L4AREXSYQVIASDM42F7ZKTOE2SNQF6YFCHDYVRWA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.133225949 (564h18m15.625658392s) [noise=true , address=45.91.116.6:4975 , piece_id=QNWEPPN57W2S47ZY5KTZX2ZUSD6M3KDKQBUMBNF2EKQJM7XTCJHA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.133584397 (564h18m15.626016842s) [noise=true , address=145.79.227.17:4971 , piece_id=AB7XCAMGPL36IB6WRHTH5XY6ZEOOJ32TA5QIRLTQWPGSAH5GOSEA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.133791056 (564h18m15.626223491s) [noise=true , address=77.90.31.21:4879 , piece_id=T7Q6EUKUDBHZI3Z4Q3GKLRNZ52PTAXIRZSXHPJP7M2AAC2FBAOKA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.133830189 (564h18m15.62626263s) [noise=true , address=77.90.31.19:4841 , piece_id=IPBSWQ2UNTOSCZVPBFYTXTBLTDX62DG23HOS5ZIUSPUQZYYCEJ3A, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134272151 (564h18m15.626704602s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZANWLER2XJNUIQLCHW4JJVH4CAFAJGRC36EXBRFYREHF2JGCKUQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134387252 (564h18m15.626819704s) [noise=true , address=145.79.227.3:4149 , piece_id=PSPWHDROT3J6HFUGOECBU3CBAAAOXS433LJG3C4L5X3HKTXXGT3Q, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134625236 (564h18m15.627057681s) [noise=true , address=145.79.227.5:4221 , piece_id=OMXPV6EHTNTOVCILLK5VFYKS3EG72W77OCGN2MIYPX2OEME7OICQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134894274 (564h18m15.627326727s) [noise=true , address=45.91.116.14:4191 , piece_id=SDKNOK2IKMKZKBTMKL3233NQVPYEIABEJ5PG3PACVNELEKCROYOQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134970870 (564h18m15.627403319s) [noise=true , address=45.91.116.14:4189 , piece_id=TJHXT45J4Q2HSLC4JVIVITKCN4544ZOWRHEE6ELYAPOXA26WBPXQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.135375758 (564h18m15.627808207s) [noise=true , address=45.91.116.4:4937 , piece_id=5JMXKZF2YILQWB6EMQURVADLGWATFI7WB44ZMUUSV5Y7PITAZD5Q, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.135850292 (564h18m15.628282745s) [noise=true , address=45.91.116.7:4011 , piece_id=SBPCP22GU7G7KM766NBAOIY7VLMCOJRWDJD5GADCU54GAYTSY6FA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.138308899 (564h18m15.630741344s) [noise=true , address=45.91.116.9:4261 , piece_id=5TE632QE5G562O5C6KLPXZK2HUCJSA47NAAOVZG6PIWDRDT23ICQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.139359971 (564h18m15.631792416s) [noise=true , address=77.90.31.13:4628 , piece_id=ZBU2IPSKMGAH4ZXXOSACZB26E7HGCJIAIZGWKXGFQF252MVGJEOQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.139841188 (564h18m15.632273641s) [noise=true , address=45.91.116.11:4119 , piece_id=LH4NHVPQ5RECQ6I44DJ4DYZ4HCJLUPUWHBWBSYLM5AZYPBWUT7UQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.140461019 (564h18m15.632893471s) [noise=true , address=145.79.227.11:4729 , piece_id=R5ZC22DU3C6SRFBJGCMBBP5DDWMTZBK3S43JBLHANZNXTZVP2YGQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.743424738 (564h18m16.235857167s) [upload=496433]: Next returned nil segment Sep 22 02:25:55.743441472 (564h18m16.2358739s) [upload=496433]: Waiting for error group managing segments... Sep 22 02:25:55.883508500 (564h18m16.375940942s) [noise=true , address=45.91.116.6:4975 , piece_id=QNWEPPN57W2S47ZY5KTZX2ZUSD6M3KDKQBUMBNF2EKQJM7XTCJHA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.883610617 (564h18m16.376043067s) [noise=true , address=45.91.116.11:4119 , piece_id=LH4NHVPQ5RECQ6I44DJ4DYZ4HCJLUPUWHBWBSYLM5AZYPBWUT7UQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.884102370 (564h18m16.376534803s) [noise=true , address=77.90.31.19:4841 , piece_id=IPBSWQ2UNTOSCZVPBFYTXTBLTDX62DG23HOS5ZIUSPUQZYYCEJ3A, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.884283835 (564h18m16.376716268s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZANWLER2XJNUIQLCHW4JJVH4CAFAJGRC36EXBRFYREHF2JGCKUQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.884530653 (564h18m16.376963087s) [noise=true , address=45.91.116.9:4261 , piece_id=5TE632QE5G562O5C6KLPXZK2HUCJSA47NAAOVZG6PIWDRDT23ICQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.885074685 (564h18m16.377507117s) [noise=true , address=45.91.116.14:4191 , piece_id=SDKNOK2IKMKZKBTMKL3233NQVPYEIABEJ5PG3PACVNELEKCROYOQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.885688933 (564h18m16.378121368s) [noise=true , address=45.91.116.4:4937 , piece_id=5JMXKZF2YILQWB6EMQURVADLGWATFI7WB44ZMUUSV5Y7PITAZD5Q, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.896172630 (564h18m16.388605059s) [noise=true , address=45.91.116.14:4189 , piece_id=TJHXT45J4Q2HSLC4JVIVITKCN4544ZOWRHEE6ELYAPOXA26WBPXQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.896200094 (564h18m16.38863253s) [noise=true , address=77.90.31.21:4879 , piece_id=T7Q6EUKUDBHZI3Z4Q3GKLRNZ52PTAXIRZSXHPJP7M2AAC2FBAOKA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.897339199 (564h18m16.389771628s) [noise=true , address=77.90.31.13:4628 , piece_id=ZBU2IPSKMGAH4ZXXOSACZB26E7HGCJIAIZGWKXGFQF252MVGJEOQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.906099406 (564h18m16.398531839s) [noise=true , address=45.91.116.7:4011 , piece_id=SBPCP22GU7G7KM766NBAOIY7VLMCOJRWDJD5GADCU54GAYTSY6FA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:26:24.734742361 (564h18m45.227174811s) [noise=true , address=145.79.227.3:4149 , piece_id=PSPWHDROT3J6HFUGOECBU3CBAAAOXS433LJG3C4L5X3HKTXXGT3Q, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.734977415 (564h18m45.227409864s) [noise=true , address=145.79.227.11:4729 , piece_id=R5ZC22DU3C6SRFBJGCMBBP5DDWMTZBK3S43JBLHANZNXTZVP2YGQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.735050621 (564h18m45.227483073s) [noise=true , address=145.79.227.17:4971 , piece_id=AB7XCAMGPL36IB6WRHTH5XY6ZEOOJ32TA5QIRLTQWPGSAH5GOSEA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.735076212 (564h18m45.227508654s) [noise=true , address=145.79.227.7:4295 , piece_id=O42XY6PSEQS2L4AREXSYQVIASDM42F7ZKTOE2SNQF6YFCHDYVRWA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.735126748 (564h18m45.227559189s) [noise=true , address=145.79.227.5:4221 , piece_id=OMXPV6EHTNTOVCILLK5VFYKS3EG72W77OCGN2MIYPX2OEME7OICQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.735160107 (564h18m45.22759254s) [seg_pos={0 0}, upload=496433]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 02:26:24.841616203 (564h18m45.334048638s) [upload=496433]: Done uploading segments... Sep 22 02:26:24.841648820 (564h18m45.334081254s) [upload=496433]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.481726627s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X.X.......X..X] duration: 30.001141751s FAIL (2025-09-22 02:26:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001189638s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....................................XXXXXXX..X.X...XXXXXXX.X] duration: 30.10928562s FAIL (2025-09-22 02:26:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 02:25:55.020025511 (564h18m15.512457939s) [upload=496433]: Starting upload Sep 22 02:25:55.020111359 (564h18m15.512543804s) [upload=496433]: Done starting upload Sep 22 02:25:55.020234446 (564h18m15.512666881s) [upload=496433]: Uploading segments... Sep 22 02:25:55.020416850 (564h18m15.512849284s) [upload=496433]: Got next segment. Inline: false Sep 22 02:25:55.020465495 (564h18m15.512897949s) [upload=496433]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:25:55.020474934 (564h18m15.51290738s) [upload=496433]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:25:55.133086277 (564h18m15.625518729s) [seg_pos={0 0}, upload=496433]: Begin upload segment... Sep 22 02:25:55.133165552 (564h18m15.625597996s) [seg_pos={0 0}, upload=496433]: Done begin upload segment. Sep 22 02:25:55.133210341 (564h18m15.625642792s) [noise=true , address=145.79.227.7:4295 , piece_id=O42XY6PSEQS2L4AREXSYQVIASDM42F7ZKTOE2SNQF6YFCHDYVRWA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.133225949 (564h18m15.625658392s) [noise=true , address=45.91.116.6:4975 , piece_id=QNWEPPN57W2S47ZY5KTZX2ZUSD6M3KDKQBUMBNF2EKQJM7XTCJHA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.133584397 (564h18m15.626016842s) [noise=true , address=145.79.227.17:4971 , piece_id=AB7XCAMGPL36IB6WRHTH5XY6ZEOOJ32TA5QIRLTQWPGSAH5GOSEA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.133791056 (564h18m15.626223491s) [noise=true , address=77.90.31.21:4879 , piece_id=T7Q6EUKUDBHZI3Z4Q3GKLRNZ52PTAXIRZSXHPJP7M2AAC2FBAOKA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.133830189 (564h18m15.62626263s) [noise=true , address=77.90.31.19:4841 , piece_id=IPBSWQ2UNTOSCZVPBFYTXTBLTDX62DG23HOS5ZIUSPUQZYYCEJ3A, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134272151 (564h18m15.626704602s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZANWLER2XJNUIQLCHW4JJVH4CAFAJGRC36EXBRFYREHF2JGCKUQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134387252 (564h18m15.626819704s) [noise=true , address=145.79.227.3:4149 , piece_id=PSPWHDROT3J6HFUGOECBU3CBAAAOXS433LJG3C4L5X3HKTXXGT3Q, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134625236 (564h18m15.627057681s) [noise=true , address=145.79.227.5:4221 , piece_id=OMXPV6EHTNTOVCILLK5VFYKS3EG72W77OCGN2MIYPX2OEME7OICQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134894274 (564h18m15.627326727s) [noise=true , address=45.91.116.14:4191 , piece_id=SDKNOK2IKMKZKBTMKL3233NQVPYEIABEJ5PG3PACVNELEKCROYOQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.134970870 (564h18m15.627403319s) [noise=true , address=45.91.116.14:4189 , piece_id=TJHXT45J4Q2HSLC4JVIVITKCN4544ZOWRHEE6ELYAPOXA26WBPXQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.135375758 (564h18m15.627808207s) [noise=true , address=45.91.116.4:4937 , piece_id=5JMXKZF2YILQWB6EMQURVADLGWATFI7WB44ZMUUSV5Y7PITAZD5Q, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.135850292 (564h18m15.628282745s) [noise=true , address=45.91.116.7:4011 , piece_id=SBPCP22GU7G7KM766NBAOIY7VLMCOJRWDJD5GADCU54GAYTSY6FA, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.138308899 (564h18m15.630741344s) [noise=true , address=45.91.116.9:4261 , piece_id=5TE632QE5G562O5C6KLPXZK2HUCJSA47NAAOVZG6PIWDRDT23ICQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.139359971 (564h18m15.631792416s) [noise=true , address=77.90.31.13:4628 , piece_id=ZBU2IPSKMGAH4ZXXOSACZB26E7HGCJIAIZGWKXGFQF252MVGJEOQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.139841188 (564h18m15.632273641s) [noise=true , address=45.91.116.11:4119 , piece_id=LH4NHVPQ5RECQ6I44DJ4DYZ4HCJLUPUWHBWBSYLM5AZYPBWUT7UQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.140461019 (564h18m15.632893471s) [noise=true , address=145.79.227.11:4729 , piece_id=R5ZC22DU3C6SRFBJGCMBBP5DDWMTZBK3S43JBLHANZNXTZVP2YGQ, seg_pos={0 0}, upload=496433]: Uploading piece... Sep 22 02:25:55.743424738 (564h18m16.235857167s) [upload=496433]: Next returned nil segment Sep 22 02:25:55.743441472 (564h18m16.2358739s) [upload=496433]: Waiting for error group managing segments... Sep 22 02:25:55.883508500 (564h18m16.375940942s) [noise=true , address=45.91.116.6:4975 , piece_id=QNWEPPN57W2S47ZY5KTZX2ZUSD6M3KDKQBUMBNF2EKQJM7XTCJHA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.883610617 (564h18m16.376043067s) [noise=true , address=45.91.116.11:4119 , piece_id=LH4NHVPQ5RECQ6I44DJ4DYZ4HCJLUPUWHBWBSYLM5AZYPBWUT7UQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.884102370 (564h18m16.376534803s) [noise=true , address=77.90.31.19:4841 , piece_id=IPBSWQ2UNTOSCZVPBFYTXTBLTDX62DG23HOS5ZIUSPUQZYYCEJ3A, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.884283835 (564h18m16.376716268s) [noise=true , address=77.90.31.10:4517 , piece_id=6ZANWLER2XJNUIQLCHW4JJVH4CAFAJGRC36EXBRFYREHF2JGCKUQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.884530653 (564h18m16.376963087s) [noise=true , address=45.91.116.9:4261 , piece_id=5TE632QE5G562O5C6KLPXZK2HUCJSA47NAAOVZG6PIWDRDT23ICQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.885074685 (564h18m16.377507117s) [noise=true , address=45.91.116.14:4191 , piece_id=SDKNOK2IKMKZKBTMKL3233NQVPYEIABEJ5PG3PACVNELEKCROYOQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.885688933 (564h18m16.378121368s) [noise=true , address=45.91.116.4:4937 , piece_id=5JMXKZF2YILQWB6EMQURVADLGWATFI7WB44ZMUUSV5Y7PITAZD5Q, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.896172630 (564h18m16.388605059s) [noise=true , address=45.91.116.14:4189 , piece_id=TJHXT45J4Q2HSLC4JVIVITKCN4544ZOWRHEE6ELYAPOXA26WBPXQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.896200094 (564h18m16.38863253s) [noise=true , address=77.90.31.21:4879 , piece_id=T7Q6EUKUDBHZI3Z4Q3GKLRNZ52PTAXIRZSXHPJP7M2AAC2FBAOKA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.897339199 (564h18m16.389771628s) [noise=true , address=77.90.31.13:4628 , piece_id=ZBU2IPSKMGAH4ZXXOSACZB26E7HGCJIAIZGWKXGFQF252MVGJEOQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:25:55.906099406 (564h18m16.398531839s) [noise=true , address=45.91.116.7:4011 , piece_id=SBPCP22GU7G7KM766NBAOIY7VLMCOJRWDJD5GADCU54GAYTSY6FA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: <nil> Sep 22 02:26:24.734742361 (564h18m45.227174811s) [noise=true , address=145.79.227.3:4149 , piece_id=PSPWHDROT3J6HFUGOECBU3CBAAAOXS433LJG3C4L5X3HKTXXGT3Q, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: rpc: dial tcp 145.79.227.3:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.734977415 (564h18m45.227409864s) [noise=true , address=145.79.227.11:4729 , piece_id=R5ZC22DU3C6SRFBJGCMBBP5DDWMTZBK3S43JBLHANZNXTZVP2YGQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:145.79.227.11:4729): protocol: rpc: dial tcp 145.79.227.11:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.735050621 (564h18m45.227483073s) [noise=true , address=145.79.227.17:4971 , piece_id=AB7XCAMGPL36IB6WRHTH5XY6ZEOOJ32TA5QIRLTQWPGSAH5GOSEA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: rpc: dial tcp 145.79.227.17:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.735076212 (564h18m45.227508654s) [noise=true , address=145.79.227.7:4295 , piece_id=O42XY6PSEQS2L4AREXSYQVIASDM42F7ZKTOE2SNQF6YFCHDYVRWA, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.735126748 (564h18m45.227559189s) [noise=true , address=145.79.227.5:4221 , piece_id=OMXPV6EHTNTOVCILLK5VFYKS3EG72W77OCGN2MIYPX2OEME7OICQ, seg_pos={0 0}, upload=496433]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:26:24.735160107 (564h18m45.22759254s) [seg_pos={0 0}, upload=496433]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 02:26:24.841616203 (564h18m45.334048638s) [upload=496433]: Done uploading segments... Sep 22 02:26:24.841648820 (564h18m45.334081254s) [upload=496433]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.109315979s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................XXXX.X....X..............] duration: 2.636377086s FAIL (2025-09-22 02:26:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.636408489s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X.X.......X..] duration: 2.959030141s FAIL (2025-09-22 02:25:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.959179815s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................XXXX.X....X.............] duration: 2.599890696s FAIL (2025-09-22 02:25:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.599916288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.X.......X.] duration: 2.703222282s FAIL (2025-09-22 02:25:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.703322625s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....................................XXXXXXX..X.X...XXXXXXX.] duration: 2.610204694s FAIL (2025-09-22 02:25:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:24:37.975789329 (564h16m58.468221775s) [upload=496411]: Starting upload Sep 22 02:24:37.975918379 (564h16m58.468350821s) [upload=496411]: Done starting upload Sep 22 02:24:37.976060291 (564h16m58.468492733s) [upload=496411]: Uploading segments... Sep 22 02:24:37.976334490 (564h16m58.468766933s) [upload=496411]: Got next segment. Inline: false Sep 22 02:24:37.976403201 (564h16m58.468835647s) [upload=496411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:24:37.976411181 (564h16m58.468843627s) [upload=496411]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:24:38.101269370 (564h16m58.593701823s) [seg_pos={0 0}, upload=496411]: Begin upload segment... Sep 22 02:24:38.101333664 (564h16m58.593766106s) [seg_pos={0 0}, upload=496411]: Done begin upload segment. Sep 22 02:24:38.101359823 (564h16m58.593792267s) [noise=true , address=77.90.31.12:4590 , piece_id=RQ4NCPCQNUQQGPEE3EYEVEAGLIAJ3OSTK662NOI6RCM2KBID5DIA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.101373840 (564h16m58.593806282s) [noise=true , address=45.91.116.7:4011 , piece_id=W7OFVE4EVVXYC2CST4YQPZXZRKSIS3SBCS2IZALXLO2DDRFM4EJQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.101791460 (564h16m58.594223908s) [noise=true , address=77.90.31.17:4771 , piece_id=XDBXGWFEG4UNPEQW4OHM5XTQAEVDZMXZBP42CQ5BVIG3R2HCPQYA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.102298525 (564h16m58.594730977s) [noise=true , address=145.79.227.16:4910 , piece_id=FCBRRW3RD2AMKTSJW3UBAFPXIRBPFG3BBWQ54WG7OWONYG45ERAQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.102458188 (564h16m58.594890636s) [noise=true , address=145.79.227.13:4801 , piece_id=P2EH3SRTIUQK4WP2IL44XIW5GCBIU7342RKU3GB4T7WVP6NJ2J6A, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.102727424 (564h16m58.595159869s) [noise=true , address=45.91.116.5:4297 , piece_id=LZ2VXJYUZYZLTSCBKEU7MGRMNM5IAJY7OAIL3SKUGO3I2WPRKG6Q, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.102824381 (564h16m58.595256824s) [noise=true , address=45.91.116.5:4299 , piece_id=H2RXC3EDUP32H3MBHU7H2GQIXFO6RSS252VBUUETKQTBPFPGEPTQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.103157520 (564h16m58.595589966s) [noise=true , address=45.91.116.7:4009 , piece_id=ANNWSLPAA6ELU6NSX3ARJI23C43IGDE5DPEN765SHARF7I5EKGQA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.103468752 (564h16m58.595901197s) [noise=true , address=77.90.31.15:4699 , piece_id=BNSPLJUC76BXCECK2IOVF6HMW3BWIEJDMV42XC4J7DOMEWP2F3VQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.103796006 (564h16m58.596228451s) [noise=true , address=45.91.116.12:4226 , piece_id=VGSZ6WBA4K6DUIZXSK6J6DEG6YLCRUECBEQ5I35EBHYLGO26TT3Q, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.104130189 (564h16m58.596562638s) [noise=true , address=145.79.227.18:4007 , piece_id=KMMPS2AZUYJLNATJB5N474ITYDPQZV52YPJWXR6LZSJYEAUFRLPQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.104496610 (564h16m58.596929052s) [noise=true , address=45.91.116.4:4938 , piece_id=LONWUJD64TBB4VPYW2JJDIOXMBSIHY75V5T4JRD3AIQARJ44U3LA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.104711273 (564h16m58.597143714s) [noise=true , address=77.90.31.21:4878 , piece_id=62VFDN4BFEY2K2RLAEA26KLLJNB46QRZRYGEPZPVJE2AEKCUQHOA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.105053633 (564h16m58.597486087s) [noise=true , address=77.90.31.18:4807 , piece_id=V3DENSGQYC7EJBQ2SMCGJE6CESOB5SZXXQGYPHUFECZ7R4IYNWXQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.105395840 (564h16m58.597828283s) [noise=true , address=45.91.116.6:4975 , piece_id=RWUVXJACFKHB65AYPJ5NPVB77CTA4T4W25MTRMQXJW3URUXIO6MQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.105509998 (564h16m58.597942446s) [noise=true , address=145.79.227.7:4293 , piece_id=765VRD6DJHW3PX74X7Q2KZOKTPU4SEH6DWQK6WJFZ7YNBXCGQQWQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.707692153 (564h16m59.200124587s) [upload=496411]: Next returned nil segment Sep 22 02:24:38.707709802 (564h16m59.200142244s) [upload=496411]: Waiting for error group managing segments... Sep 22 02:24:38.847766124 (564h16m59.340198559s) [noise=true , address=45.91.116.7:4011 , piece_id=W7OFVE4EVVXYC2CST4YQPZXZRKSIS3SBCS2IZALXLO2DDRFM4EJQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.847819712 (564h16m59.340252155s) [noise=true , address=77.90.31.15:4699 , piece_id=BNSPLJUC76BXCECK2IOVF6HMW3BWIEJDMV42XC4J7DOMEWP2F3VQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.848264703 (564h16m59.340697132s) [noise=true , address=45.91.116.6:4975 , piece_id=RWUVXJACFKHB65AYPJ5NPVB77CTA4T4W25MTRMQXJW3URUXIO6MQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.848298958 (564h16m59.340731386s) [noise=true , address=77.90.31.21:4878 , piece_id=62VFDN4BFEY2K2RLAEA26KLLJNB46QRZRYGEPZPVJE2AEKCUQHOA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.849464711 (564h16m59.341897145s) [noise=true , address=45.91.116.5:4299 , piece_id=H2RXC3EDUP32H3MBHU7H2GQIXFO6RSS252VBUUETKQTBPFPGEPTQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.849696043 (564h16m59.342128475s) [noise=true , address=77.90.31.12:4590 , piece_id=RQ4NCPCQNUQQGPEE3EYEVEAGLIAJ3OSTK662NOI6RCM2KBID5DIA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.850226575 (564h16m59.342659014s) [noise=true , address=77.90.31.17:4771 , piece_id=XDBXGWFEG4UNPEQW4OHM5XTQAEVDZMXZBP42CQ5BVIG3R2HCPQYA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.850292490 (564h16m59.342724921s) [noise=true , address=45.91.116.12:4226 , piece_id=VGSZ6WBA4K6DUIZXSK6J6DEG6YLCRUECBEQ5I35EBHYLGO26TT3Q, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.851590366 (564h16m59.344022796s) [noise=true , address=45.91.116.7:4009 , piece_id=ANNWSLPAA6ELU6NSX3ARJI23C43IGDE5DPEN765SHARF7I5EKGQA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.859143067 (564h16m59.351575497s) [noise=true , address=45.91.116.4:4938 , piece_id=LONWUJD64TBB4VPYW2JJDIOXMBSIHY75V5T4JRD3AIQARJ44U3LA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.864351026 (564h16m59.356783456s) [noise=true , address=77.90.31.18:4807 , piece_id=V3DENSGQYC7EJBQ2SMCGJE6CESOB5SZXXQGYPHUFECZ7R4IYNWXQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.865165791 (564h16m59.357598227s) [noise=true , address=45.91.116.5:4297 , piece_id=LZ2VXJYUZYZLTSCBKEU7MGRMNM5IAJY7OAIL3SKUGO3I2WPRKG6Q, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:25:07.675817038 (564h17m28.168249482s) [noise=true , address=145.79.227.7:4293 , piece_id=765VRD6DJHW3PX74X7Q2KZOKTPU4SEH6DWQK6WJFZ7YNBXCGQQWQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:25:07.675905275 (564h17m28.168337723s) [noise=true , address=145.79.227.13:4801 , piece_id=P2EH3SRTIUQK4WP2IL44XIW5GCBIU7342RKU3GB4T7WVP6NJ2J6A, seg_pos={0 0}, upload=496411]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:25:07.675992567 (564h17m28.168425012s) [noise=true , address=145.79.227.16:4910 , piece_id=FCBRRW3RD2AMKTSJW3UBAFPXIRBPFG3BBWQ54WG7OWONYG45ERAQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:25:07.676109928 (564h17m28.16854237s) [noise=true , address=145.79.227.18:4007 , piece_id=KMMPS2AZUYJLNATJB5N474ITYDPQZV52YPJWXR6LZSJYEAUFRLPQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:25:07.676162833 (564h17m28.168595287s) [seg_pos={0 0}, upload=496411]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:25:07.782252458 (564h17m28.274684901s) [upload=496411]: Done uploading segments... Sep 22 02:25:07.782279542 (564h17m28.27471199s) [upload=496411]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.610228436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......................................XXXXXXX..X.X...XXXXXXX] duration: 30.10829206s FAIL (2025-09-22 02:25:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:24:37.975789329 (564h16m58.468221775s) [upload=496411]: Starting upload Sep 22 02:24:37.975918379 (564h16m58.468350821s) [upload=496411]: Done starting upload Sep 22 02:24:37.976060291 (564h16m58.468492733s) [upload=496411]: Uploading segments... Sep 22 02:24:37.976334490 (564h16m58.468766933s) [upload=496411]: Got next segment. Inline: false Sep 22 02:24:37.976403201 (564h16m58.468835647s) [upload=496411]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:24:37.976411181 (564h16m58.468843627s) [upload=496411]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:24:38.101269370 (564h16m58.593701823s) [seg_pos={0 0}, upload=496411]: Begin upload segment... Sep 22 02:24:38.101333664 (564h16m58.593766106s) [seg_pos={0 0}, upload=496411]: Done begin upload segment. Sep 22 02:24:38.101359823 (564h16m58.593792267s) [noise=true , address=77.90.31.12:4590 , piece_id=RQ4NCPCQNUQQGPEE3EYEVEAGLIAJ3OSTK662NOI6RCM2KBID5DIA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.101373840 (564h16m58.593806282s) [noise=true , address=45.91.116.7:4011 , piece_id=W7OFVE4EVVXYC2CST4YQPZXZRKSIS3SBCS2IZALXLO2DDRFM4EJQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.101791460 (564h16m58.594223908s) [noise=true , address=77.90.31.17:4771 , piece_id=XDBXGWFEG4UNPEQW4OHM5XTQAEVDZMXZBP42CQ5BVIG3R2HCPQYA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.102298525 (564h16m58.594730977s) [noise=true , address=145.79.227.16:4910 , piece_id=FCBRRW3RD2AMKTSJW3UBAFPXIRBPFG3BBWQ54WG7OWONYG45ERAQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.102458188 (564h16m58.594890636s) [noise=true , address=145.79.227.13:4801 , piece_id=P2EH3SRTIUQK4WP2IL44XIW5GCBIU7342RKU3GB4T7WVP6NJ2J6A, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.102727424 (564h16m58.595159869s) [noise=true , address=45.91.116.5:4297 , piece_id=LZ2VXJYUZYZLTSCBKEU7MGRMNM5IAJY7OAIL3SKUGO3I2WPRKG6Q, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.102824381 (564h16m58.595256824s) [noise=true , address=45.91.116.5:4299 , piece_id=H2RXC3EDUP32H3MBHU7H2GQIXFO6RSS252VBUUETKQTBPFPGEPTQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.103157520 (564h16m58.595589966s) [noise=true , address=45.91.116.7:4009 , piece_id=ANNWSLPAA6ELU6NSX3ARJI23C43IGDE5DPEN765SHARF7I5EKGQA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.103468752 (564h16m58.595901197s) [noise=true , address=77.90.31.15:4699 , piece_id=BNSPLJUC76BXCECK2IOVF6HMW3BWIEJDMV42XC4J7DOMEWP2F3VQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.103796006 (564h16m58.596228451s) [noise=true , address=45.91.116.12:4226 , piece_id=VGSZ6WBA4K6DUIZXSK6J6DEG6YLCRUECBEQ5I35EBHYLGO26TT3Q, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.104130189 (564h16m58.596562638s) [noise=true , address=145.79.227.18:4007 , piece_id=KMMPS2AZUYJLNATJB5N474ITYDPQZV52YPJWXR6LZSJYEAUFRLPQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.104496610 (564h16m58.596929052s) [noise=true , address=45.91.116.4:4938 , piece_id=LONWUJD64TBB4VPYW2JJDIOXMBSIHY75V5T4JRD3AIQARJ44U3LA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.104711273 (564h16m58.597143714s) [noise=true , address=77.90.31.21:4878 , piece_id=62VFDN4BFEY2K2RLAEA26KLLJNB46QRZRYGEPZPVJE2AEKCUQHOA, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.105053633 (564h16m58.597486087s) [noise=true , address=77.90.31.18:4807 , piece_id=V3DENSGQYC7EJBQ2SMCGJE6CESOB5SZXXQGYPHUFECZ7R4IYNWXQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.105395840 (564h16m58.597828283s) [noise=true , address=45.91.116.6:4975 , piece_id=RWUVXJACFKHB65AYPJ5NPVB77CTA4T4W25MTRMQXJW3URUXIO6MQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.105509998 (564h16m58.597942446s) [noise=true , address=145.79.227.7:4293 , piece_id=765VRD6DJHW3PX74X7Q2KZOKTPU4SEH6DWQK6WJFZ7YNBXCGQQWQ, seg_pos={0 0}, upload=496411]: Uploading piece... Sep 22 02:24:38.707692153 (564h16m59.200124587s) [upload=496411]: Next returned nil segment Sep 22 02:24:38.707709802 (564h16m59.200142244s) [upload=496411]: Waiting for error group managing segments... Sep 22 02:24:38.847766124 (564h16m59.340198559s) [noise=true , address=45.91.116.7:4011 , piece_id=W7OFVE4EVVXYC2CST4YQPZXZRKSIS3SBCS2IZALXLO2DDRFM4EJQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.847819712 (564h16m59.340252155s) [noise=true , address=77.90.31.15:4699 , piece_id=BNSPLJUC76BXCECK2IOVF6HMW3BWIEJDMV42XC4J7DOMEWP2F3VQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.848264703 (564h16m59.340697132s) [noise=true , address=45.91.116.6:4975 , piece_id=RWUVXJACFKHB65AYPJ5NPVB77CTA4T4W25MTRMQXJW3URUXIO6MQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.848298958 (564h16m59.340731386s) [noise=true , address=77.90.31.21:4878 , piece_id=62VFDN4BFEY2K2RLAEA26KLLJNB46QRZRYGEPZPVJE2AEKCUQHOA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.849464711 (564h16m59.341897145s) [noise=true , address=45.91.116.5:4299 , piece_id=H2RXC3EDUP32H3MBHU7H2GQIXFO6RSS252VBUUETKQTBPFPGEPTQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.849696043 (564h16m59.342128475s) [noise=true , address=77.90.31.12:4590 , piece_id=RQ4NCPCQNUQQGPEE3EYEVEAGLIAJ3OSTK662NOI6RCM2KBID5DIA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.850226575 (564h16m59.342659014s) [noise=true , address=77.90.31.17:4771 , piece_id=XDBXGWFEG4UNPEQW4OHM5XTQAEVDZMXZBP42CQ5BVIG3R2HCPQYA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.850292490 (564h16m59.342724921s) [noise=true , address=45.91.116.12:4226 , piece_id=VGSZ6WBA4K6DUIZXSK6J6DEG6YLCRUECBEQ5I35EBHYLGO26TT3Q, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.851590366 (564h16m59.344022796s) [noise=true , address=45.91.116.7:4009 , piece_id=ANNWSLPAA6ELU6NSX3ARJI23C43IGDE5DPEN765SHARF7I5EKGQA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.859143067 (564h16m59.351575497s) [noise=true , address=45.91.116.4:4938 , piece_id=LONWUJD64TBB4VPYW2JJDIOXMBSIHY75V5T4JRD3AIQARJ44U3LA, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.864351026 (564h16m59.356783456s) [noise=true , address=77.90.31.18:4807 , piece_id=V3DENSGQYC7EJBQ2SMCGJE6CESOB5SZXXQGYPHUFECZ7R4IYNWXQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:24:38.865165791 (564h16m59.357598227s) [noise=true , address=45.91.116.5:4297 , piece_id=LZ2VXJYUZYZLTSCBKEU7MGRMNM5IAJY7OAIL3SKUGO3I2WPRKG6Q, seg_pos={0 0}, upload=496411]: Done uploading piece. err: <nil> Sep 22 02:25:07.675817038 (564h17m28.168249482s) [noise=true , address=145.79.227.7:4293 , piece_id=765VRD6DJHW3PX74X7Q2KZOKTPU4SEH6DWQK6WJFZ7YNBXCGQQWQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:145.79.227.7:4293): protocol: rpc: dial tcp 145.79.227.7:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:25:07.675905275 (564h17m28.168337723s) [noise=true , address=145.79.227.13:4801 , piece_id=P2EH3SRTIUQK4WP2IL44XIW5GCBIU7342RKU3GB4T7WVP6NJ2J6A, seg_pos={0 0}, upload=496411]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:25:07.675992567 (564h17m28.168425012s) [noise=true , address=145.79.227.16:4910 , piece_id=FCBRRW3RD2AMKTSJW3UBAFPXIRBPFG3BBWQ54WG7OWONYG45ERAQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:25:07.676109928 (564h17m28.16854237s) [noise=true , address=145.79.227.18:4007 , piece_id=KMMPS2AZUYJLNATJB5N474ITYDPQZV52YPJWXR6LZSJYEAUFRLPQ, seg_pos={0 0}, upload=496411]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:25:07.676162833 (564h17m28.168595287s) [seg_pos={0 0}, upload=496411]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:25:07.782252458 (564h17m28.274684901s) [upload=496411]: Done uploading segments... Sep 22 02:25:07.782279542 (564h17m28.27471199s) [upload=496411]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.108312009s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X.X.......X] duration: 30.0010036s FAIL (2025-09-22 02:24:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001028448s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................................XXXX.X....X............] duration: 3.102043318s FAIL (2025-09-22 02:24:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 3.102070874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......................................XXXXXXX..X.X...XXXXXX] duration: 30.113498969s FAIL (2025-09-22 02:24:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 02:24:07.692252326 (564h16m28.184684775s) [upload=496403]: Starting upload Sep 22 02:24:07.692373872 (564h16m28.184806323s) [upload=496403]: Done starting upload Sep 22 02:24:07.692522094 (564h16m28.184954537s) [upload=496403]: Uploading segments... Sep 22 02:24:07.692893547 (564h16m28.185325976s) [upload=496403]: Got next segment. Inline: false Sep 22 02:24:07.692966934 (564h16m28.185399387s) [upload=496403]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:24:07.692973676 (564h16m28.185406136s) [upload=496403]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:24:07.807317911 (564h16m28.299750364s) [seg_pos={0 0}, upload=496403]: Begin upload segment... Sep 22 02:24:07.807396755 (564h16m28.299829206s) [seg_pos={0 0}, upload=496403]: Done begin upload segment. Sep 22 02:24:07.807450261 (564h16m28.299882711s) [noise=true , address=45.91.116.14:4189 , piece_id=B3UNU265ANA5XBS4TN2L2BGFZVZFV2D32QZEQUOPR3QONALDZLIQ, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.807849538 (564h16m28.300281984s) [noise=true , address=45.91.116.4:4938 , piece_id=IVPC3IOOEYEVB7JFTZ52BU4URTXFG67NAY4I35JFC6P4JP6DTEWQ, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.808051054 (564h16m28.300483496s) [noise=true , address=45.91.116.5:4298 , piece_id=TUMCR2L3XVS7HN5WXY53OV76CLRUVVZHBHJQSWCFMDHFVR676VKA, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.808459616 (564h16m28.300892059s) [noise=true , address=45.91.116.13:4155 , piece_id=OXP6HTPP5YJTBB2DLQMZ6PBIWUDEY2B63CCOUNACWLYQOAFKKWMQ, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.808541888 (564h16m28.300974329s) [noise=true , address=77.90.31.18:4806 , piece_id=JBANNOJALAFI2DXPVSIYDCZKID45YMWY5LMXYWYERDZQ3QZO67OQ, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.809038507 (564h16m28.301470949s) [noise=true , address=45.91.116.11:4118 , piece_id=PFSDV4T4EPYCP3TNGRWZA32AQ36OZMDGTM7Q66DA6VLIWKEC4BFA, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.809126892 (564h16m28.301559335s) [noise=true , address=77.90.31.11:4555 , piece_id=LVVDHPTBQY5RJU7G2AEYO7YP4M2SH2ZCS5ZNNLJN7DY2JE4O5RUA, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.809712274 (564h16m28.302144718s) [noise=true , address=77.90.31.19:4841 , piece_id=5E2HA3NTAZYSAPYBGYQRRNETFQFPIB6HXIRXNKL5KIR4LI7DSP7A, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.810085693 (564h16m28.302518143s) [noise=true , address=45.91.116.13:4154 , piece_id=CECYXLMXTWEGIZMQQMGTLNOR77FWANEG3PEVHRKYB52F5EE7VEDA, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.810553857 (564h16m28.302986299s) [noise=true , address=145.79.227.5:4222 , piece_id=3Z4UNTNFRBTVH6CJLHGTJEN5VESQLWB4LVSUWS4SM6JWZTUHLZNQ, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.810816163 (564h16m28.303248608s) [noise=true , address=145.79.227.18:4005 , piece_id=P6KMGVXHH57KEVAL3VLZIHHR6IH2UAI7DJZJT2ULYF522SR5YIGA, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.812182867 (564h16m28.30461531s) [noise=true , address=145.79.227.3:4151 , piece_id=DUUYBNBJPC4XPXPE2ZCQVNCOVDRXBM5JCW3Y2SSFES6SAXIL2W6A, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.812423025 (564h16m28.304855468s) [noise=true , address=145.79.227.15:4875 , piece_id=VYFAYJICUAW3PXNNCTRYMKLKJ3LTIDH2ZVN7RWUN36LO4DFZYA6A, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.812724839 (564h16m28.305157281s) [noise=true , address=77.90.31.12:4590 , piece_id=3ND4I6VHBLUASIWMBHUJTU5VO6NUPC3QXIRDFFTS44Q6PTIK6YPQ, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.812962419 (564h16m28.305394861s) [noise=true , address=145.79.227.14:4839 , piece_id=BIBZJRYWMUYNNMCO72SBZ7BQDGEFJHOJZQSSIWLXT4MYVVWJIQWQ, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:07.813190839 (564h16m28.305623282s) [noise=true , address=45.91.116.8:4045 , piece_id=XR3E2HW7KBUKLO457FB33BZIPO2FOS46ELDXP5I2THL7GV426VVQ, seg_pos={0 0}, upload=496403]: Uploading piece... Sep 22 02:24:08.420309305 (564h16m28.912741749s) [upload=496403]: Next returned nil segment Sep 22 02:24:08.420328344 (564h16m28.912760796s) [upload=496403]: Waiting for error group managing segments... Sep 22 02:24:08.553492349 (564h16m29.045924802s) [noise=true , address=45.91.116.4:4938 , piece_id=IVPC3IOOEYEVB7JFTZ52BU4URTXFG67NAY4I35JFC6P4JP6DTEWQ, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.554574265 (564h16m29.047006712s) [noise=true , address=77.90.31.12:4590 , piece_id=3ND4I6VHBLUASIWMBHUJTU5VO6NUPC3QXIRDFFTS44Q6PTIK6YPQ, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.554767190 (564h16m29.047199619s) [noise=true , address=45.91.116.13:4155 , piece_id=OXP6HTPP5YJTBB2DLQMZ6PBIWUDEY2B63CCOUNACWLYQOAFKKWMQ, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.555347349 (564h16m29.047779793s) [noise=true , address=45.91.116.11:4118 , piece_id=PFSDV4T4EPYCP3TNGRWZA32AQ36OZMDGTM7Q66DA6VLIWKEC4BFA, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.558023564 (564h16m29.050455992s) [noise=true , address=77.90.31.11:4555 , piece_id=LVVDHPTBQY5RJU7G2AEYO7YP4M2SH2ZCS5ZNNLJN7DY2JE4O5RUA, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.559185478 (564h16m29.051617929s) [noise=true , address=45.91.116.8:4045 , piece_id=XR3E2HW7KBUKLO457FB33BZIPO2FOS46ELDXP5I2THL7GV426VVQ, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.560209818 (564h16m29.052642262s) [noise=true , address=77.90.31.19:4841 , piece_id=5E2HA3NTAZYSAPYBGYQRRNETFQFPIB6HXIRXNKL5KIR4LI7DSP7A, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.572700659 (564h16m29.065133089s) [noise=true , address=45.91.116.13:4154 , piece_id=CECYXLMXTWEGIZMQQMGTLNOR77FWANEG3PEVHRKYB52F5EE7VEDA, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.574531663 (564h16m29.066964092s) [noise=true , address=45.91.116.14:4189 , piece_id=B3UNU265ANA5XBS4TN2L2BGFZVZFV2D32QZEQUOPR3QONALDZLIQ, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.576230098 (564h16m29.068662528s) [noise=true , address=45.91.116.5:4298 , piece_id=TUMCR2L3XVS7HN5WXY53OV76CLRUVVZHBHJQSWCFMDHFVR676VKA, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:08.579534745 (564h16m29.071967179s) [noise=true , address=77.90.31.18:4806 , piece_id=JBANNOJALAFI2DXPVSIYDCZKID45YMWY5LMXYWYERDZQ3QZO67OQ, seg_pos={0 0}, upload=496403]: Done uploading piece. err: <nil> Sep 22 02:24:37.402997840 (564h16m57.895430283s) [noise=true , address=145.79.227.15:4875 , piece_id=VYFAYJICUAW3PXNNCTRYMKLKJ3LTIDH2ZVN7RWUN36LO4DFZYA6A, seg_pos={0 0}, upload=496403]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:24:37.403026237 (564h16m57.895458688s) [noise=true , address=145.79.227.3:4151 , piece_id=DUUYBNBJPC4XPXPE2ZCQVNCOVDRXBM5JCW3Y2SSFES6SAXIL2W6A, seg_pos={0 0}, upload=496403]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:24:37.403062415 (564h16m57.895494843s) [noise=true , address=145.79.227.14:4839 , piece_id=BIBZJRYWMUYNNMCO72SBZ7BQDGEFJHOJZQSSIWLXT4MYVVWJIQWQ, seg_pos={0 0}, upload=496403]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: rpc: dial tcp 145.79.227.14:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:24:37.403098915 (564h16m57.895531358s) [noise=true , address=145.79.227.18:4005 , piece_id=P6KMGVXHH57KEVAL3VLZIHHR6IH2UAI7DJZJT2ULYF522SR5YIGA, seg_pos={0 0}, upload=496403]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:145.79.227.18:4005): protocol: rpc: dial tcp 145.79.227.18:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:24:37.403113976 (564h16m57.895546405s) [noise=true , address=145.79.227.5:4222 , piece_id=3Z4UNTNFRBTVH6CJLHGTJEN5VESQLWB4LVSUWS4SM6JWZTUHLZNQ, seg_pos={0 0}, upload=496403]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:24:37.403155791 (564h16m57.895588243s) [seg_pos={0 0}, upload=496403]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 02:24:37.514267261 (564h16m58.006699703s) [upload=496403]: Done uploading segments... Sep 22 02:24:37.514295018 (564h16m58.006727452s) [upload=496403]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.113529988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................................XXXX.X....X...........] duration: 2.797140441s FAIL (2025-09-22 02:24:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.797175016s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 2.811714248s FAIL (2025-09-22 02:23:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.811743932s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........................................XXXXXXX..X.X...XXXXX] duration: 30.118318209s FAIL (2025-09-22 02:23:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== .467577835s) [upload=496390]: Starting upload Sep 22 02:23:08.975227436 (564h15m29.467659887s) [upload=496390]: Done starting upload Sep 22 02:23:08.975647593 (564h15m29.468080029s) [upload=496390]: Uploading segments... Sep 22 02:23:08.975715332 (564h15m29.468147783s) [upload=496390]: Got next segment. Inline: false Sep 22 02:23:08.975756625 (564h15m29.468189068s) [upload=496390]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:23:08.975761848 (564h15m29.46819429s) [upload=496390]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:23:09.097518623 (564h15m29.589951055s) [seg_pos={0 0}, upload=496390]: Begin upload segment... Sep 22 02:23:09.097581298 (564h15m29.59001373s) [seg_pos={0 0}, upload=496390]: Done begin upload segment. Sep 22 02:23:09.097609113 (564h15m29.590041556s) [noise=true , address=145.79.227.17:4972 , piece_id=2T32ZN2ZZ2EKAX3RZAYVCDSJHCN7KPC35HXE3FHM3A4L4UJGA2QQ, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.097827400 (564h15m29.590259835s) [noise=true , address=77.90.31.21:4877 , piece_id=IP4HK6PXOQUOIJUNF76JPUASCIKI3N42C6Q6KJVRBHQPINEM4LTQ, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.098357772 (564h15m29.590790208s) [noise=true , address=145.79.227.5:4221 , piece_id=E7JBXIZX37MIMBXWCZJY35VTP4GZNBEJUFOBYH2H7W4L6UKWUILQ, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.099322545 (564h15m29.591754985s) [noise=true , address=145.79.227.17:4969 , piece_id=ANLO6TYBKEVBDQUPT6ZDUOCRXT57T53K5LX34BAJPJGQZKJLHEZA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.100451304 (564h15m29.592883739s) [noise=true , address=77.90.31.18:4807 , piece_id=7WZLOOSXJWZJVJEP5KUITFRLD7BTKAPTKXX562DFVT2PLPRAS4HA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.101127890 (564h15m29.593560324s) [noise=true , address=45.91.116.7:4010 , piece_id=6J7QH5CG46XSONDVI3QPQP57ODAGU3ESU7NNCYJHADFXWSHSWPSQ, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.101857854 (564h15m29.594290287s) [noise=true , address=145.79.227.11:4730 , piece_id=PM55DRYLM7XY3BRYIREYJSJHAOXATKTP4YVNVZIY2V4VI6JAWNNA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.103724404 (564h15m29.596156839s) [noise=true , address=145.79.227.2:4115 , piece_id=I77DHYHCPFJU22HPVY6XV3BOCIPKEHK6IKTRX6HRAFOFEPHOOVLQ, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.105177288 (564h15m29.597609739s) [noise=true , address=145.79.227.4:4186 , piece_id=EOMU4WSJZAUOGCMDNK3LTK6NEUYPRS6SBMH7SEHVLCDAOVLKIDQA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.106355633 (564h15m29.598788079s) [noise=true , address=145.79.227.18:4007 , piece_id=MGDAJ5MYFURIBMHCEZDOCZABR2WBFXTHEXVHN3GML4F3K4QCRPHA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.107105896 (564h15m29.599538334s) [noise=true , address=45.91.116.8:4045 , piece_id=AF52XMXUALDA253BXRCJHS7U4ZQ6YIZILPXP25RQAGEBB53DFNFA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.107871539 (564h15m29.600303974s) [noise=true , address=45.91.116.5:4297 , piece_id=5YBOQQUNV6BTPHRTBEHQ6QML2K7YEAYLACNXJYJKCSVVFX3K53EA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.107968551 (564h15m29.600400996s) [noise=true , address=145.79.227.7:4295 , piece_id=UPLAEF2EVE47MQIKDSNQIMBXISENH4G5ICY6MRDRLRYZ4IQBSSBQ, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.108537218 (564h15m29.600969654s) [noise=true , address=45.91.116.9:4261 , piece_id=LH5GGOVQYXDOGUDHARVB4RL6UUE6D7OC63SVZZA6H6YLNPC7TAUA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.109682711 (564h15m29.602115145s) [noise=true , address=77.90.31.19:4842 , piece_id=I7WLU2B7JC26DOJJH3FNZSPUYGXCAEACCWYZA7KMWVS7HONHWKXA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.111427887 (564h15m29.603860321s) [noise=true , address=77.90.31.21:4879 , piece_id=ZWDUGAO2CSJ4KT5BKQGJUQX5N5D4J7Q4IYJ2G365GKC26EMPQWBA, seg_pos={0 0}, upload=496390]: Uploading piece... Sep 22 02:23:09.714501559 (564h15m30.206933992s) [upload=496390]: Next returned nil segment Sep 22 02:23:09.714525234 (564h15m30.206957689s) [upload=496390]: Waiting for error group managing segments... Sep 22 02:23:09.847946924 (564h15m30.340379374s) [noise=true , address=45.91.116.5:4297 , piece_id=5YBOQQUNV6BTPHRTBEHQ6QML2K7YEAYLACNXJYJKCSVVFX3K53EA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: <nil> Sep 22 02:23:09.847968511 (564h15m30.340400958s) [noise=true , address=45.91.116.8:4045 , piece_id=AF52XMXUALDA253BXRCJHS7U4ZQ6YIZILPXP25RQAGEBB53DFNFA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: <nil> Sep 22 02:23:09.851070920 (564h15m30.343503367s) [noise=true , address=45.91.116.7:4010 , piece_id=6J7QH5CG46XSONDVI3QPQP57ODAGU3ESU7NNCYJHADFXWSHSWPSQ, seg_pos={0 0}, upload=496390]: Done uploading piece. err: <nil> Sep 22 02:23:09.860650236 (564h15m30.353082672s) [noise=true , address=45.91.116.9:4261 , piece_id=LH5GGOVQYXDOGUDHARVB4RL6UUE6D7OC63SVZZA6H6YLNPC7TAUA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: <nil> Sep 22 02:23:09.861533060 (564h15m30.353965508s) [noise=true , address=77.90.31.19:4842 , piece_id=I7WLU2B7JC26DOJJH3FNZSPUYGXCAEACCWYZA7KMWVS7HONHWKXA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: <nil> Sep 22 02:23:09.864141818 (564h15m30.356574253s) [noise=true , address=77.90.31.18:4807 , piece_id=7WZLOOSXJWZJVJEP5KUITFRLD7BTKAPTKXX562DFVT2PLPRAS4HA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: <nil> Sep 22 02:23:09.870331630 (564h15m30.362764059s) [noise=true , address=77.90.31.21:4877 , piece_id=IP4HK6PXOQUOIJUNF76JPUASCIKI3N42C6Q6KJVRBHQPINEM4LTQ, seg_pos={0 0}, upload=496390]: Done uploading piece. err: <nil> Sep 22 02:23:09.880705549 (564h15m30.373137979s) [noise=true , address=77.90.31.21:4879 , piece_id=ZWDUGAO2CSJ4KT5BKQGJUQX5N5D4J7Q4IYJ2G365GKC26EMPQWBA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: <nil> Sep 22 02:23:38.672999897 (564h15m59.165432325s) [noise=true , address=145.79.227.17:4969 , piece_id=ANLO6TYBKEVBDQUPT6ZDUOCRXT57T53K5LX34BAJPJGQZKJLHEZA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:145.79.227.17:4969): protocol: rpc: dial tcp 145.79.227.17:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:38.673051459 (564h15m59.165483887s) [noise=true , address=145.79.227.11:4730 , piece_id=PM55DRYLM7XY3BRYIREYJSJHAOXATKTP4YVNVZIY2V4VI6JAWNNA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:38.673075319 (564h15m59.165507748s) [noise=true , address=145.79.227.4:4186 , piece_id=EOMU4WSJZAUOGCMDNK3LTK6NEUYPRS6SBMH7SEHVLCDAOVLKIDQA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:38.673107514 (564h15m59.165539941s) [noise=true , address=145.79.227.5:4221 , piece_id=E7JBXIZX37MIMBXWCZJY35VTP4GZNBEJUFOBYH2H7W4L6UKWUILQ, seg_pos={0 0}, upload=496390]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:38.673125674 (564h15m59.165558102s) [noise=true , address=145.79.227.17:4972 , piece_id=2T32ZN2ZZ2EKAX3RZAYVCDSJHCN7KPC35HXE3FHM3A4L4UJGA2QQ, seg_pos={0 0}, upload=496390]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:38.673143954 (564h15m59.165576383s) [noise=true , address=145.79.227.2:4115 , piece_id=I77DHYHCPFJU22HPVY6XV3BOCIPKEHK6IKTRX6HRAFOFEPHOOVLQ, seg_pos={0 0}, upload=496390]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:38.673174928 (564h15m59.165607357s) [noise=true , address=145.79.227.7:4295 , piece_id=UPLAEF2EVE47MQIKDSNQIMBXISENH4G5ICY6MRDRLRYZ4IQBSSBQ, seg_pos={0 0}, upload=496390]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: rpc: dial tcp 145.79.227.7:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:38.673194185 (564h15m59.165626613s) [noise=true , address=145.79.227.18:4007 , piece_id=MGDAJ5MYFURIBMHCEZDOCZABR2WBFXTHEXVHN3GML4F3K4QCRPHA, seg_pos={0 0}, upload=496390]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: rpc: dial tcp 145.79.227.18:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:38.673210083 (564h15m59.165642536s) [seg_pos={0 0}, upload=496390]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 02:23:38.787479565 (564h15m59.279912023s) [upload=496390]: Done uploading segments... Sep 22 02:23:38.788417474 (564h15m59.280849919s) [upload=496390]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.118335925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................................XXXX.X....X..........] duration: 2.608316074s FAIL (2025-09-22 02:23:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.608348874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........................................XXXXXXX..X.X...XXXX] duration: 30.111675026s FAIL (2025-09-22 02:23:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:22:38.699702764 (564h14m59.192135197s) [upload=496383]: Starting upload Sep 22 02:22:38.699807942 (564h14m59.192240387s) [upload=496383]: Done starting upload Sep 22 02:22:38.700006867 (564h14m59.19243931s) [upload=496383]: Uploading segments... Sep 22 02:22:38.700253410 (564h14m59.192685839s) [upload=496383]: Got next segment. Inline: false Sep 22 02:22:38.700306694 (564h14m59.192739136s) [upload=496383]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:22:38.700311432 (564h14m59.192743875s) [upload=496383]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:22:38.887820999 (564h14m59.38025345s) [seg_pos={0 0}, upload=496383]: Begin upload segment... Sep 22 02:22:38.887885404 (564h14m59.380317853s) [seg_pos={0 0}, upload=496383]: Done begin upload segment. Sep 22 02:22:38.887918381 (564h14m59.380350834s) [noise=true , address=145.79.227.2:4115 , piece_id=EB6I6H2YBWGBAZNSTL765JZJC3O66F34AGEN4OQJBZ326AHKQG3A, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.887955464 (564h14m59.380387915s) [noise=true , address=145.79.227.11:4730 , piece_id=ER4Y2BOKZYGIHSVI3URFITLLNERRVR5WJWRJIY5JC4S3AXCECHZA, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.888298214 (564h14m59.380730662s) [noise=true , address=77.90.31.10:4517 , piece_id=ZUYMT6Q7ZW62KSB7T4DLZCAFWTO7DHGPOAXMR3WCP5GSNUWVUWQA, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.888371305 (564h14m59.380803747s) [noise=true , address=77.90.31.13:4627 , piece_id=AB5PQX3DD2RQIP7NKYMCB4FRDMA6XFWXAEUZAOK3SNSZCX3FXO6A, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.888535629 (564h14m59.380968074s) [noise=true , address=45.91.116.4:4937 , piece_id=WQGVRBEGWQB4TBVZNTEBJIPQ6RZOT7WWYF2DJ2JOY5NQBZK7KXMA, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.888692093 (564h14m59.381124538s) [noise=true , address=45.91.116.8:4047 , piece_id=BXKWTUYJDZORL5O3XJ55DWWKRGCUQZ75PHXSEQJXR3OPMXWNDWMQ, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.888776073 (564h14m59.381208517s) [noise=true , address=45.91.116.13:4155 , piece_id=ESFY7I3U6L27X3R5PWF4JDIQBKIOJBJFZPIUI3TFWHVTXRV3FU7Q, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.889005400 (564h14m59.381437845s) [noise=true , address=45.91.116.5:4297 , piece_id=J5IHXKNJA2YPZRDYUWGW2YEOJGH4ZU6GHDZ4BB3TK6JKBBEROWAQ, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.889099546 (564h14m59.381531995s) [noise=true , address=45.91.116.11:4119 , piece_id=BYS7T7FBDOLTRTWBBFIXFDBICQE7OBWALMAUMDZ2YYFIUY7NPJKA, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.889232353 (564h14m59.381664799s) [noise=true , address=77.90.31.14:4663 , piece_id=OYDDJJMVB5SN6D5NKW2PKOHQVDC4WQVZRSFPAMSNG6VDDUZOK7IQ, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.889468971 (564h14m59.381901415s) [noise=true , address=45.91.116.12:4225 , piece_id=CS7VPLA7AQBDOKPOW47QK3EESFFHWWIG2IEJGXFIOGEXECGEPVJA, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.889571142 (564h14m59.382003594s) [noise=true , address=77.90.31.15:4699 , piece_id=UTE7LJXTCRX4EYORLQQEEAL65GLI32DXJOVRW3N57AND7IWPFGQA, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.889702867 (564h14m59.382135309s) [noise=true , address=145.79.227.16:4910 , piece_id=4PRYZKW3RUWRHAT62MV4ALCVFLMVT5BAACWA2V2WIWFSYQ4NPSEA, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.890062570 (564h14m59.382495026s) [noise=true , address=77.90.31.21:4878 , piece_id=5P7X224RDPBFI7ORF3A2ROQPWTWCDXQXJT5KZTESYLVUK2QPDTJQ, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.892531578 (564h14m59.384964025s) [noise=true , address=145.79.227.6:4258 , piece_id=B6GPJLYKQ7RN7GQYZNEWJTMJDVAJWRMGSYSYHNSVFWHMJ5W7N2YQ, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:38.892847753 (564h14m59.385280201s) [noise=true , address=77.90.31.21:4879 , piece_id=AQJPX3NJGDCUGSRD4R5BAUACOIEOQWLI2HEJHETCJQQQ3V3X356Q, seg_pos={0 0}, upload=496383]: Uploading piece... Sep 22 02:22:39.503715726 (564h14m59.996148167s) [upload=496383]: Next returned nil segment Sep 22 02:22:39.503736246 (564h14m59.996168692s) [upload=496383]: Waiting for error group managing segments... Sep 22 02:22:39.644889014 (564h15m0.137321442s) [noise=true , address=45.91.116.5:4297 , piece_id=J5IHXKNJA2YPZRDYUWGW2YEOJGH4ZU6GHDZ4BB3TK6JKBBEROWAQ, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.647100606 (564h15m0.139533042s) [noise=true , address=77.90.31.13:4627 , piece_id=AB5PQX3DD2RQIP7NKYMCB4FRDMA6XFWXAEUZAOK3SNSZCX3FXO6A, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.647356645 (564h15m0.139789082s) [noise=true , address=77.90.31.10:4517 , piece_id=ZUYMT6Q7ZW62KSB7T4DLZCAFWTO7DHGPOAXMR3WCP5GSNUWVUWQA, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.647608150 (564h15m0.140040582s) [noise=true , address=45.91.116.12:4225 , piece_id=CS7VPLA7AQBDOKPOW47QK3EESFFHWWIG2IEJGXFIOGEXECGEPVJA, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.648845501 (564h15m0.141277929s) [noise=true , address=45.91.116.13:4155 , piece_id=ESFY7I3U6L27X3R5PWF4JDIQBKIOJBJFZPIUI3TFWHVTXRV3FU7Q, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.649954152 (564h15m0.142386581s) [noise=true , address=77.90.31.14:4663 , piece_id=OYDDJJMVB5SN6D5NKW2PKOHQVDC4WQVZRSFPAMSNG6VDDUZOK7IQ, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.650474049 (564h15m0.142906484s) [noise=true , address=45.91.116.4:4937 , piece_id=WQGVRBEGWQB4TBVZNTEBJIPQ6RZOT7WWYF2DJ2JOY5NQBZK7KXMA, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.653838002 (564h15m0.146270437s) [noise=true , address=77.90.31.15:4699 , piece_id=UTE7LJXTCRX4EYORLQQEEAL65GLI32DXJOVRW3N57AND7IWPFGQA, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.653932633 (564h15m0.146365068s) [noise=true , address=77.90.31.21:4878 , piece_id=5P7X224RDPBFI7ORF3A2ROQPWTWCDXQXJT5KZTESYLVUK2QPDTJQ, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.655898343 (564h15m0.148330786s) [noise=true , address=77.90.31.21:4879 , piece_id=AQJPX3NJGDCUGSRD4R5BAUACOIEOQWLI2HEJHETCJQQQ3V3X356Q, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.657395644 (564h15m0.149828079s) [noise=true , address=45.91.116.11:4119 , piece_id=BYS7T7FBDOLTRTWBBFIXFDBICQE7OBWALMAUMDZ2YYFIUY7NPJKA, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:22:39.660457521 (564h15m0.152889949s) [noise=true , address=45.91.116.8:4047 , piece_id=BXKWTUYJDZORL5O3XJ55DWWKRGCUQZ75PHXSEQJXR3OPMXWNDWMQ, seg_pos={0 0}, upload=496383]: Done uploading piece. err: <nil> Sep 22 02:23:08.398662370 (564h15m28.891094819s) [noise=true , address=145.79.227.11:4730 , piece_id=ER4Y2BOKZYGIHSVI3URFITLLNERRVR5WJWRJIY5JC4S3AXCECHZA, seg_pos={0 0}, upload=496383]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:145.79.227.11:4730): protocol: rpc: dial tcp 145.79.227.11:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:08.398718738 (564h15m28.891151181s) [noise=true , address=145.79.227.2:4115 , piece_id=EB6I6H2YBWGBAZNSTL765JZJC3O66F34AGEN4OQJBZ326AHKQG3A, seg_pos={0 0}, upload=496383]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:08.398761068 (564h15m28.891193498s) [noise=true , address=145.79.227.16:4910 , piece_id=4PRYZKW3RUWRHAT62MV4ALCVFLMVT5BAACWA2V2WIWFSYQ4NPSEA, seg_pos={0 0}, upload=496383]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: rpc: dial tcp 145.79.227.16:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:08.398768363 (564h15m28.891200793s) [noise=true , address=145.79.227.6:4258 , piece_id=B6GPJLYKQ7RN7GQYZNEWJTMJDVAJWRMGSYSYHNSVFWHMJ5W7N2YQ, seg_pos={0 0}, upload=496383]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:23:08.398803478 (564h15m28.891235931s) [seg_pos={0 0}, upload=496383]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:23:08.508238301 (564h15m29.000670733s) [upload=496383]: Done uploading segments... Sep 22 02:23:08.508295574 (564h15m29.000728024s) [upload=496383]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.111695523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................................XXXX.X....X.........] duration: 2.525166176s FAIL (2025-09-22 02:22:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.525195727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 2.791033141s FAIL (2025-09-22 02:22:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.791094629s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........................................XXXXXXX..X.X...XXX] duration: 30.120918406s FAIL (2025-09-22 02:22:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 02:22:08.427474022 (564h14m28.919906471s) [upload=496373]: Starting upload Sep 22 02:22:08.427589691 (564h14m28.920022141s) [upload=496373]: Done starting upload Sep 22 02:22:08.427703508 (564h14m28.920135973s) [upload=496373]: Uploading segments... Sep 22 02:22:08.428030919 (564h14m28.92046335s) [upload=496373]: Got next segment. Inline: false Sep 22 02:22:08.428105958 (564h14m28.920538484s) [upload=496373]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:22:08.428112926 (564h14m28.920545376s) [upload=496373]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:22:08.548603907 (564h14m29.041036357s) [seg_pos={0 0}, upload=496373]: Begin upload segment... Sep 22 02:22:08.548663140 (564h14m29.041095593s) [seg_pos={0 0}, upload=496373]: Done begin upload segment. Sep 22 02:22:08.548694426 (564h14m29.041126878s) [noise=true , address=145.79.227.6:4257 , piece_id=FWXKTRBTKQRQEYR5Z4MU2BRAJLBOYACOF6MXCJMNG5A5VOJJ2VYA, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.549204069 (564h14m29.041636522s) [noise=true , address=45.91.116.4:4938 , piece_id=AQDMW3KRSFLMTCTD5PSAWSIG2HSV5YQLT4QETBOJOJBEFUTJC3AQ, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.549614247 (564h14m29.042046696s) [noise=true , address=45.91.116.5:4299 , piece_id=BYRWCQDOHRBUIRCTPDZZYWDFMAIPJKWPGPIMBY6E7XSHKM6L5LAA, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.550052766 (564h14m29.042485218s) [noise=true , address=45.91.116.5:4297 , piece_id=CXTAZKMWD7ZSCOYBVJVX3AKKV636U3HMMSB6DHRYF43BSO5MSRUA, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.550292579 (564h14m29.042725024s) [noise=true , address=145.79.227.13:4801 , piece_id=SX2PHKCLFZSKGN5RLT4NZXTFQAHTJA7AIT2O6VJRQK56MCCI6C3A, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.550877176 (564h14m29.043309629s) [noise=true , address=77.90.31.13:4627 , piece_id=PYW5NF4AAAUAVJPJR3KPL7IHNOQUI6JIBJ4DSFESTZVIFZDCYF6A, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.551297392 (564h14m29.043729837s) [noise=true , address=45.91.116.10:4082 , piece_id=AZAIGZF7OKIRCJCRT23OAZZ6TGG4DSZRIVIBAOITULI6WH56RTHA, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.553571992 (564h14m29.046004444s) [noise=true , address=45.91.116.13:4153 , piece_id=ASREA7JK2O6PXMBS4WRUJAM6JRU6H4YQ6NILOL4IZRP7QCNTWACA, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.554415823 (564h14m29.046848269s) [noise=true , address=45.91.116.11:4117 , piece_id=XMPV7BPUHP3LPVJKPWZDMVEKJFGI2DBUKP5WNGVGALC4YXSK2WPA, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.555341623 (564h14m29.047774074s) [noise=true , address=145.79.227.18:4006 , piece_id=MGHGGFBMG7EUMFSGX7W7MRNAMNNWDI3DSJBGWADVL7EA33WOMN7Q, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.556149848 (564h14m29.048582307s) [noise=true , address=45.91.116.6:4975 , piece_id=ZIYB7DYQWKGZ3NNHCSNOO4C6EL7BGD6WB6NALHFXWGDHFIDG4CSQ, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.557586801 (564h14m29.050019247s) [noise=true , address=77.90.31.21:4878 , piece_id=AN6CQ6ZRWL2TWEYKJ5KBMATBLJU67OFPSNRCTEGZDD7PBK2JFAUA, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.558039193 (564h14m29.050471646s) [noise=true , address=77.90.31.10:4517 , piece_id=QUJBLTTI2USP5SFWZCPM3BZQFPO72DNM3QLZKMWLQMMMCAHEDTBA, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.558507653 (564h14m29.050940102s) [noise=true , address=145.79.227.14:4837 , piece_id=ASYASUJGVX3XOXDWJKAYSANJG6DDN7FSWRZQIRULL4TFNHKORM6Q, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.558841348 (564h14m29.051273796s) [noise=true , address=77.90.31.19:4842 , piece_id=54WWY77WKQ2ENVZ4DWR3DXQ33EYTF6DW6NYXQ6MS3G5OBSTOVFFQ, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:08.559166101 (564h14m29.051598549s) [noise=true , address=145.79.227.7:4294 , piece_id=BD4TFUPN5UW5S4GOPRNXZTBTB6P4YK4X6CVPM5UAVFEVWEYFOYLQ, seg_pos={0 0}, upload=496373]: Uploading piece... Sep 22 02:22:09.167497068 (564h14m29.659929501s) [upload=496373]: Next returned nil segment Sep 22 02:22:09.167521432 (564h14m29.659953882s) [upload=496373]: Waiting for error group managing segments... Sep 22 02:22:09.298587128 (564h14m29.791019574s) [noise=true , address=45.91.116.11:4117 , piece_id=XMPV7BPUHP3LPVJKPWZDMVEKJFGI2DBUKP5WNGVGALC4YXSK2WPA, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.298696601 (564h14m29.791129035s) [noise=true , address=45.91.116.5:4299 , piece_id=BYRWCQDOHRBUIRCTPDZZYWDFMAIPJKWPGPIMBY6E7XSHKM6L5LAA, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.304386543 (564h14m29.796818989s) [noise=true , address=45.91.116.10:4082 , piece_id=AZAIGZF7OKIRCJCRT23OAZZ6TGG4DSZRIVIBAOITULI6WH56RTHA, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.306030619 (564h14m29.798463048s) [noise=true , address=77.90.31.13:4627 , piece_id=PYW5NF4AAAUAVJPJR3KPL7IHNOQUI6JIBJ4DSFESTZVIFZDCYF6A, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.306386511 (564h14m29.798818954s) [noise=true , address=45.91.116.5:4297 , piece_id=CXTAZKMWD7ZSCOYBVJVX3AKKV636U3HMMSB6DHRYF43BSO5MSRUA, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.310606952 (564h14m29.803039386s) [noise=true , address=77.90.31.10:4517 , piece_id=QUJBLTTI2USP5SFWZCPM3BZQFPO72DNM3QLZKMWLQMMMCAHEDTBA, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.312154227 (564h14m29.804586671s) [noise=true , address=45.91.116.13:4153 , piece_id=ASREA7JK2O6PXMBS4WRUJAM6JRU6H4YQ6NILOL4IZRP7QCNTWACA, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.312524651 (564h14m29.804957086s) [noise=true , address=45.91.116.4:4938 , piece_id=AQDMW3KRSFLMTCTD5PSAWSIG2HSV5YQLT4QETBOJOJBEFUTJC3AQ, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.312671877 (564h14m29.80510432s) [noise=true , address=45.91.116.6:4975 , piece_id=ZIYB7DYQWKGZ3NNHCSNOO4C6EL7BGD6WB6NALHFXWGDHFIDG4CSQ, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.321164715 (564h14m29.813597147s) [noise=true , address=77.90.31.19:4842 , piece_id=54WWY77WKQ2ENVZ4DWR3DXQ33EYTF6DW6NYXQ6MS3G5OBSTOVFFQ, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:09.324732284 (564h14m29.817164713s) [noise=true , address=77.90.31.21:4878 , piece_id=AN6CQ6ZRWL2TWEYKJ5KBMATBLJU67OFPSNRCTEGZDD7PBK2JFAUA, seg_pos={0 0}, upload=496373]: Done uploading piece. err: <nil> Sep 22 02:22:38.124991332 (564h14m58.617423777s) [noise=true , address=145.79.227.6:4257 , piece_id=FWXKTRBTKQRQEYR5Z4MU2BRAJLBOYACOF6MXCJMNG5A5VOJJ2VYA, seg_pos={0 0}, upload=496373]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:145.79.227.6:4257): protocol: rpc: dial tcp 145.79.227.6:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:22:38.125075153 (564h14m58.617507597s) [noise=true , address=145.79.227.18:4006 , piece_id=MGHGGFBMG7EUMFSGX7W7MRNAMNNWDI3DSJBGWADVL7EA33WOMN7Q, seg_pos={0 0}, upload=496373]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:22:38.125139483 (564h14m58.617572037s) [noise=true , address=145.79.227.13:4801 , piece_id=SX2PHKCLFZSKGN5RLT4NZXTFQAHTJA7AIT2O6VJRQK56MCCI6C3A, seg_pos={0 0}, upload=496373]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:22:38.125200112 (564h14m58.617632547s) [noise=true , address=145.79.227.14:4837 , piece_id=ASYASUJGVX3XOXDWJKAYSANJG6DDN7FSWRZQIRULL4TFNHKORM6Q, seg_pos={0 0}, upload=496373]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:22:38.125258846 (564h14m58.617691279s) [noise=true , address=145.79.227.7:4294 , piece_id=BD4TFUPN5UW5S4GOPRNXZTBTB6P4YK4X6CVPM5UAVFEVWEYFOYLQ, seg_pos={0 0}, upload=496373]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:22:38.125278458 (564h14m58.61771091s) [seg_pos={0 0}, upload=496373]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 02:22:38.237958640 (564h14m58.730391087s) [upload=496373]: Done uploading segments... Sep 22 02:22:38.237984848 (564h14m58.730417298s) [upload=496373]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.120939615s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 2.7454844s FAIL (2025-09-22 02:22:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.745521622s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................................XXXX.X....X........] duration: 2.600673511s FAIL (2025-09-22 02:22:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.600785732s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...........................................XXXXXXX..X.X...XX] duration: 30.135953456s FAIL (2025-09-22 02:22:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 02:21:38.124875486 (564h13m58.61730794s) [upload=496366]: Starting upload Sep 22 02:21:38.124982928 (564h13m58.617415365s) [upload=496366]: Done starting upload Sep 22 02:21:38.125148815 (564h13m58.617581265s) [upload=496366]: Uploading segments... Sep 22 02:21:38.125339157 (564h13m58.6177716s) [upload=496366]: Got next segment. Inline: false Sep 22 02:21:38.125397895 (564h13m58.617830345s) [upload=496366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:21:38.125404822 (564h13m58.617837266s) [upload=496366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:21:38.261578254 (564h13m58.754010715s) [seg_pos={0 0}, upload=496366]: Begin upload segment... Sep 22 02:21:38.261667746 (564h13m58.754100179s) [seg_pos={0 0}, upload=496366]: Done begin upload segment. Sep 22 02:21:38.261717720 (564h13m58.754150221s) [noise=true , address=145.79.227.13:4801 , piece_id=W5ITGVAKKO534SFG7XPFGWSEFCVYXWOUZIV6TC4T36S53O327LNA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.262340816 (564h13m58.754773269s) [noise=true , address=145.79.227.7:4294 , piece_id=BTKNUNMUU5H6TAVMVVNRTGJCKJPQC2IVTDI5453VP5CCYYGMCTOA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.263188593 (564h13m58.755621025s) [noise=true , address=77.90.31.12:4590 , piece_id=FDZIYKRH5A5YUJ3N3AOB5NCK7JAC2WW5VNKIORTNDFIEU7JBHU3Q, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.263236462 (564h13m58.755668918s) [noise=true , address=145.79.227.12:4767 , piece_id=PPXNF4JFBKE6JPVMHRSUNQ36EZRAMT2RXFKVZFRYVZCWZV33BE6Q, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.263650348 (564h13m58.756082793s) [noise=true , address=45.91.116.13:4154 , piece_id=ZTUTJBTXQZHDOI5BOKEWCO35B5DID4GFLJWQPZAXXT45JTIIW3NQ, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.264180085 (564h13m58.756612535s) [noise=true , address=77.90.31.13:4627 , piece_id=YZZXYCAAAXJNFF6TBAG7KJSKQMHSEMB55UAFN6GIHQTGV7PDV4DA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.264564340 (564h13m58.75699677s) [noise=true , address=45.91.116.7:4010 , piece_id=Q3CS7KEJQWWCK3MUCPKY33JPZE5TE5XBRA7H3O3HHR7YLKQDVNEA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.265515616 (564h13m58.757948062s) [noise=true , address=77.90.31.19:4842 , piece_id=H24YTYZ475V564YCDG7ILYAOBGJ3UTRWF6I6PQ7ZL3NP2LMXL3TA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.265639253 (564h13m58.758071708s) [noise=true , address=45.91.116.11:4119 , piece_id=VBGZ3UE37BJ5HO4R46K76ISDMFVJCCRD5RXDTHW7EBOPWABVWDSA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.266570888 (564h13m58.75900334s) [noise=true , address=45.91.116.12:4226 , piece_id=HUISJWR3IG3RT5TZK4FZPIIGTV6OK3BDEPIUG2Q6STR32Q35DGTA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.266649606 (564h13m58.759082054s) [noise=true , address=77.90.31.18:4806 , piece_id=GGNWBA335RFWQVZN3DCKBNV6T25I7UXNLZUNBHKEPLSTPQ7JZVDA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.266973211 (564h13m58.759405652s) [noise=true , address=77.90.31.14:4661 , piece_id=DCWM5FL7KJN3LLF34WYLFNFHLFFCZUTDHKQ422VWUCRYWVTHM36Q, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.267553906 (564h13m58.759986354s) [noise=true , address=45.91.116.14:4189 , piece_id=DLIZ4MVHUW75SU2XIUKBNRWVCAMXLYYBISSBYC24FV5JY4OZBP4A, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.268047036 (564h13m58.760479482s) [noise=true , address=77.90.31.10:4518 , piece_id=DHF4YTQMQ6CQRSTGJNVXN43EPDHMDJO3TBJIU3JWZILZHYLH35UA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.268339913 (564h13m58.760772359s) [noise=true , address=145.79.227.14:4837 , piece_id=HDZPWLWPLFS4CILBNPWRI5GA7HGEZSQOX6FFQ2BZUDZTWFQOHRPA, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.268607212 (564h13m58.76103966s) [noise=true , address=45.91.116.10:4081 , piece_id=UQAPT6Z3DVGQCEA56VOL35ANIHRYTLQJHMZHJTJ2L2MGA4IFZCWQ, seg_pos={0 0}, upload=496366]: Uploading piece... Sep 22 02:21:38.887936152 (564h13m59.380368588s) [upload=496366]: Next returned nil segment Sep 22 02:21:38.887958861 (564h13m59.380391306s) [upload=496366]: Waiting for error group managing segments... Sep 22 02:21:39.025764247 (564h13m59.51819668s) [noise=true , address=45.91.116.7:4010 , piece_id=Q3CS7KEJQWWCK3MUCPKY33JPZE5TE5XBRA7H3O3HHR7YLKQDVNEA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.027599455 (564h13m59.520031884s) [noise=true , address=77.90.31.12:4590 , piece_id=FDZIYKRH5A5YUJ3N3AOB5NCK7JAC2WW5VNKIORTNDFIEU7JBHU3Q, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.027672518 (564h13m59.520104961s) [noise=true , address=77.90.31.14:4661 , piece_id=DCWM5FL7KJN3LLF34WYLFNFHLFFCZUTDHKQ422VWUCRYWVTHM36Q, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.028933001 (564h13m59.521365435s) [noise=true , address=45.91.116.13:4154 , piece_id=ZTUTJBTXQZHDOI5BOKEWCO35B5DID4GFLJWQPZAXXT45JTIIW3NQ, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.029181631 (564h13m59.521614073s) [noise=true , address=45.91.116.14:4189 , piece_id=DLIZ4MVHUW75SU2XIUKBNRWVCAMXLYYBISSBYC24FV5JY4OZBP4A, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.029618112 (564h13m59.522050557s) [noise=true , address=45.91.116.10:4081 , piece_id=UQAPT6Z3DVGQCEA56VOL35ANIHRYTLQJHMZHJTJ2L2MGA4IFZCWQ, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.029846845 (564h13m59.52227928s) [noise=true , address=45.91.116.12:4226 , piece_id=HUISJWR3IG3RT5TZK4FZPIIGTV6OK3BDEPIUG2Q6STR32Q35DGTA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.031887602 (564h13m59.524320031s) [noise=true , address=77.90.31.10:4518 , piece_id=DHF4YTQMQ6CQRSTGJNVXN43EPDHMDJO3TBJIU3JWZILZHYLH35UA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.034310151 (564h13m59.526742596s) [noise=true , address=77.90.31.18:4806 , piece_id=GGNWBA335RFWQVZN3DCKBNV6T25I7UXNLZUNBHKEPLSTPQ7JZVDA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.034906069 (564h13m59.527338511s) [noise=true , address=77.90.31.13:4627 , piece_id=YZZXYCAAAXJNFF6TBAG7KJSKQMHSEMB55UAFN6GIHQTGV7PDV4DA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.035318321 (564h13m59.52775075s) [noise=true , address=77.90.31.19:4842 , piece_id=H24YTYZ475V564YCDG7ILYAOBGJ3UTRWF6I6PQ7ZL3NP2LMXL3TA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:21:39.040692077 (564h13m59.533124525s) [noise=true , address=45.91.116.11:4119 , piece_id=VBGZ3UE37BJ5HO4R46K76ISDMFVJCCRD5RXDTHW7EBOPWABVWDSA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: <nil> Sep 22 02:22:07.843548708 (564h14m28.335981146s) [noise=true , address=145.79.227.14:4837 , piece_id=HDZPWLWPLFS4CILBNPWRI5GA7HGEZSQOX6FFQ2BZUDZTWFQOHRPA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: rpc: dial tcp 145.79.227.14:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:22:07.843583154 (564h14m28.336015591s) [noise=true , address=145.79.227.7:4294 , piece_id=BTKNUNMUU5H6TAVMVVNRTGJCKJPQC2IVTDI5453VP5CCYYGMCTOA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:145.79.227.7:4294): protocol: rpc: dial tcp 145.79.227.7:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:22:07.843607067 (564h14m28.336039519s) [noise=true , address=145.79.227.12:4767 , piece_id=PPXNF4JFBKE6JPVMHRSUNQ36EZRAMT2RXFKVZFRYVZCWZV33BE6Q, seg_pos={0 0}, upload=496366]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:22:07.843646297 (564h14m28.336078741s) [noise=true , address=145.79.227.13:4801 , piece_id=W5ITGVAKKO534SFG7XPFGWSEFCVYXWOUZIV6TC4T36S53O327LNA, seg_pos={0 0}, upload=496366]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:22:07.843684241 (564h14m28.336116685s) [seg_pos={0 0}, upload=496366]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 02:22:07.956629019 (564h14m28.449061472s) [upload=496366]: Done uploading segments... Sep 22 02:22:07.956663651 (564h14m28.449096094s) [upload=496366]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.135981453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................................XXXX.X....X.......] duration: 2.544566817s FAIL (2025-09-22 02:21:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.544685962s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 2.818317555s FAIL (2025-09-22 02:21:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.818362944s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............................................XXXXXXX..X.X...X] duration: 30.116414499s FAIL (2025-09-22 02:21:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 22 02:20:33.964665063 (564h12m54.457097492s) [upload=496351]: Starting upload Sep 22 02:20:33.964724860 (564h12m54.457157302s) [upload=496351]: Done starting upload Sep 22 02:20:33.964860496 (564h12m54.457292946s) [upload=496351]: Uploading segments... Sep 22 02:20:33.965026249 (564h12m54.457458709s) [upload=496351]: Got next segment. Inline: false Sep 22 02:20:33.965076262 (564h12m54.457508719s) [upload=496351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:20:33.965082476 (564h12m54.457514927s) [upload=496351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:20:34.089983119 (564h12m54.582415566s) [seg_pos={0 0}, upload=496351]: Begin upload segment... Sep 22 02:20:34.090042429 (564h12m54.582474864s) [seg_pos={0 0}, upload=496351]: Done begin upload segment. Sep 22 02:20:34.090068065 (564h12m54.582500507s) [noise=true , address=145.79.227.18:4006 , piece_id=7WOUU44CYZAPTMCLZUEVBVNS7IKF56WNGHPOY6ZROWTEIH26OQNA, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.090104800 (564h12m54.582537264s) [noise=true , address=45.91.116.4:4937 , piece_id=P2NLFREXTBILA6DMVW5SNXLZLZKAS4PGUCL36TZIE63IWOHVDYBQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.090583295 (564h12m54.583015743s) [noise=true , address=145.79.227.6:4259 , piece_id=S2J75BZJSP3NCZA7D3R5JXAFEAXVJGRRTBYRQBR45ML6HSEJUTCQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.090592476 (564h12m54.583024928s) [noise=true , address=45.91.116.13:4155 , piece_id=2ZWERGX5CTZHHQVHMJUFHBDBUMQRAHIUCULCPHDDSQR56IWZVRWA, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.091475179 (564h12m54.583907629s) [noise=true , address=145.79.227.5:4223 , piece_id=4HTNIBP7AW7CHX5KOU5BC4DHTBZX4BV7YGW5LZB55SBRDKA3J7DA, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.091897790 (564h12m54.584330241s) [noise=true , address=77.90.31.18:4805 , piece_id=NMPB3VULUOVIJOSPOHC2NYAF32ZXYTPXKPOW5XQQZWGDPAOISNOQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.092221574 (564h12m54.58465402s) [noise=true , address=45.91.116.5:4299 , piece_id=766VEVTNGQUVPY2TE7DA72XCEMQCQM7BR47CQ5LRJMHRVKPST46A, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.093212824 (564h12m54.585645289s) [noise=true , address=145.79.227.13:4801 , piece_id=NVNWKGZSE5EMLF5Z45VOEKOQMKL5NLX3XPNULVGEQTFCRWC764MQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.093426410 (564h12m54.585858855s) [noise=true , address=45.91.116.11:4119 , piece_id=4F7QUEXEGVEVPO64MR5XQ4QPIU7RINFCOCPOE2BRX4SYYNBYZHEQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.094598894 (564h12m54.587031346s) [noise=true , address=45.91.116.14:4190 , piece_id=6OHRZETNIG3KKQG24AUYMWNPC6QEVTWBTEJ3WJXS2HHFVSW3F6LQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.094810140 (564h12m54.587242592s) [noise=true , address=77.90.31.19:4841 , piece_id=76ADT4JW7LFBH7QTM4XN67RVCSL55HYESDCG55E3EC75LAH5ARRQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.095640175 (564h12m54.58807263s) [noise=true , address=77.90.31.10:4520 , piece_id=HEDF6G252YUHL3ABLF54X3NACQGNDAYIJDNMLYJYFXGNEQTANOFQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.096016103 (564h12m54.588448549s) [noise=true , address=45.91.116.13:4154 , piece_id=36BI4DWCFR3NADVVZUYNEIEHAXTINBH54GDCG2FEPXU5SP33TWSQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.096128643 (564h12m54.588561095s) [noise=true , address=145.79.227.11:4732 , piece_id=WATAV66NJQW6RD5QFUMGCLCJQ3LDJ37YUBYAQTRGYYCY3VU5PSDQ, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.096428313 (564h12m54.588860755s) [noise=true , address=45.91.116.10:4083 , piece_id=S3ULITKY3U3HHKA3BRPTB6YHJP6BWKD5U2BZ55ZTTWTKMH3D6SKA, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.096805015 (564h12m54.589237459s) [noise=true , address=77.90.31.15:4699 , piece_id=GAH35URZONDRYN35FH4IYCE7XKQ7XOS6ARWQ2OM5DVHUCHR7L2JA, seg_pos={0 0}, upload=496351]: Uploading piece... Sep 22 02:20:34.705856997 (564h12m55.198289449s) [upload=496351]: Next returned nil segment Sep 22 02:20:34.705930387 (564h12m55.198362831s) [upload=496351]: Waiting for error group managing segments... Sep 22 02:20:34.843021038 (564h12m55.335453473s) [noise=true , address=77.90.31.18:4805 , piece_id=NMPB3VULUOVIJOSPOHC2NYAF32ZXYTPXKPOW5XQQZWGDPAOISNOQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.843611396 (564h12m55.336043829s) [noise=true , address=45.91.116.4:4937 , piece_id=P2NLFREXTBILA6DMVW5SNXLZLZKAS4PGUCL36TZIE63IWOHVDYBQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.849096797 (564h12m55.34152923s) [noise=true , address=45.91.116.5:4299 , piece_id=766VEVTNGQUVPY2TE7DA72XCEMQCQM7BR47CQ5LRJMHRVKPST46A, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.849791948 (564h12m55.342224379s) [noise=true , address=45.91.116.11:4119 , piece_id=4F7QUEXEGVEVPO64MR5XQ4QPIU7RINFCOCPOE2BRX4SYYNBYZHEQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.852300974 (564h12m55.344733423s) [noise=true , address=77.90.31.19:4841 , piece_id=76ADT4JW7LFBH7QTM4XN67RVCSL55HYESDCG55E3EC75LAH5ARRQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.864325385 (564h12m55.356757826s) [noise=true , address=45.91.116.13:4155 , piece_id=2ZWERGX5CTZHHQVHMJUFHBDBUMQRAHIUCULCPHDDSQR56IWZVRWA, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.864411244 (564h12m55.356843678s) [noise=true , address=45.91.116.10:4083 , piece_id=S3ULITKY3U3HHKA3BRPTB6YHJP6BWKD5U2BZ55ZTTWTKMH3D6SKA, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.864992421 (564h12m55.357424854s) [noise=true , address=45.91.116.14:4190 , piece_id=6OHRZETNIG3KKQG24AUYMWNPC6QEVTWBTEJ3WJXS2HHFVSW3F6LQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.868924454 (564h12m55.361356892s) [noise=true , address=45.91.116.13:4154 , piece_id=36BI4DWCFR3NADVVZUYNEIEHAXTINBH54GDCG2FEPXU5SP33TWSQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.872126231 (564h12m55.36455866s) [noise=true , address=77.90.31.15:4699 , piece_id=GAH35URZONDRYN35FH4IYCE7XKQ7XOS6ARWQ2OM5DVHUCHR7L2JA, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:20:34.876658187 (564h12m55.369090655s) [noise=true , address=77.90.31.10:4520 , piece_id=HEDF6G252YUHL3ABLF54X3NACQGNDAYIJDNMLYJYFXGNEQTANOFQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: <nil> Sep 22 02:21:03.664909631 (564h13m24.157342083s) [noise=true , address=145.79.227.11:4732 , piece_id=WATAV66NJQW6RD5QFUMGCLCJQ3LDJ37YUBYAQTRGYYCY3VU5PSDQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: rpc: dial tcp 145.79.227.11:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:21:03.665014582 (564h13m24.157447028s) [noise=true , address=145.79.227.13:4801 , piece_id=NVNWKGZSE5EMLF5Z45VOEKOQMKL5NLX3XPNULVGEQTFCRWC764MQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:145.79.227.13:4801): protocol: rpc: dial tcp 145.79.227.13:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:21:03.665203513 (564h13m24.157635958s) [noise=true , address=145.79.227.5:4223 , piece_id=4HTNIBP7AW7CHX5KOU5BC4DHTBZX4BV7YGW5LZB55SBRDKA3J7DA, seg_pos={0 0}, upload=496351]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:21:03.665394815 (564h13m24.157827264s) [noise=true , address=145.79.227.6:4259 , piece_id=S2J75BZJSP3NCZA7D3R5JXAFEAXVJGRRTBYRQBR45ML6HSEJUTCQ, seg_pos={0 0}, upload=496351]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:145.79.227.6:4259): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:21:03.665642427 (564h13m24.158074938s) [noise=true , address=145.79.227.18:4006 , piece_id=7WOUU44CYZAPTMCLZUEVBVNS7IKF56WNGHPOY6ZROWTEIH26OQNA, seg_pos={0 0}, upload=496351]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:21:03.665671487 (564h13m24.158103971s) [seg_pos={0 0}, upload=496351]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 22 02:21:03.777963374 (564h13m24.270395824s) [upload=496351]: Done uploading segments... Sep 22 02:21:03.778002511 (564h13m24.270434967s) [upload=496351]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.116448564s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................................XXXX.X....X......] duration: 2.642041061s FAIL (2025-09-22 02:20:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.642073276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 2.940423001s FAIL (2025-09-22 02:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.940453681s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................................XXXX.X....X.....] duration: 2.50654426s FAIL (2025-09-22 02:19:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.506576897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 2.788549256s FAIL (2025-09-22 02:19:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.788570683s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................................XXXX.X....X....] duration: 2.656005314s FAIL (2025-09-22 02:19:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0159s) [seg_pos={0 0}, upload=496292]: Begin upload segment... Sep 22 02:16:16.443075073 (564h8m36.935507503s) [seg_pos={0 0}, upload=496292]: Done begin upload segment. Sep 22 02:16:16.443105441 (564h8m36.935537893s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.443347197 (564h8m36.935779648s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444152041 (564h8m36.93658457s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444600279 (564h8m36.937032731s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.444883603 (564h8m36.937316047s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445112196 (564h8m36.937544647s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.445956555 (564h8m36.938389007s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.446278645 (564h8m36.938711093s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447121359 (564h8m36.939553811s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447247470 (564h8m36.939679921s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.447700559 (564h8m36.940133007s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448088869 (564h8m36.940521323s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448540353 (564h8m36.940972802s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448647427 (564h8m36.941079875s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448905693 (564h8m36.941338139s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.448985663 (564h8m36.941418114s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.449364887 (564h8m36.941797333s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:16.450441809 (564h8m36.942874258s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Uploading piece... Sep 22 02:16:17.016630581 (564h8m37.509063022s) [upload=496292]: Next returned nil segment Sep 22 02:16:17.016651972 (564h8m37.509084415s) [upload=496292]: Waiting for error group managing segments... Sep 22 02:16:17.163540560 (564h8m37.655973015s) [noise=true , address=77.90.31.19:4871 , piece_id=TB3Z4XV77ZKBEG3E4MJMUZIJ6XKJM4ILIBARRFU7XIUV2KO4UGPQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.171809778 (564h8m37.664242211s) [noise=true , address=45.91.116.12:4232 , piece_id=WIKBPCGQXX7FGG4N3FJT36FPQ5TVJS5UP43J7K77WZSQIFMTZLVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.195795139 (564h8m37.688227574s) [noise=true , address=145.79.227.31:4831 , piece_id=MVSUPUPNT7EI4PMXWGNZ75ROQ62C5NQLIJONWKRBSYMFBR5XVVUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.198518034 (564h8m37.690950469s) [noise=true , address=145.79.227.21:4500 , piece_id=KBLWTMLGDKXN2HDK3PKXMVDIFSOCG6JORCH7TW7NWMYZPGSTUCKA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.205267668 (564h8m37.697700103s) [noise=true , address=145.79.227.26:4680 , piece_id=FQELAKU4S5ISTJD3CQG6TXWLS5G7KB42ED7XHFQTMEE5QMWYVSTQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.212402984 (564h8m37.704835418s) [noise=true , address=145.79.227.45:4260 , piece_id=EOCQ2LDBF2TB63PQEK5QIVIUEY6XJIWJ63Y5D3IFGKWQJIC7VAVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.216238221 (564h8m37.708670657s) [noise=true , address=145.79.227.52:4467 , piece_id=VPYBXZVAYVYYIW55NCDDZW5JIB4UUDBGW26YPDE4P6OEKOL3RVMQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.218275922 (564h8m37.710708358s) [noise=true , address=145.79.227.33:4900 , piece_id=2FDOCBNM3CRDKCKMWQPP6U6FBWQ7FUNS7CTITSH52L5EWQ4WENVQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.222720118 (564h8m37.715152552s) [noise=true , address=145.79.227.24:4591 , piece_id=UGHKZVNIPSTIMSX7SOXU2WBZMYFGAN7P524KGNR2B6GQXYMREFOA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.234576123 (564h8m37.727008552s) [noise=true , address=145.79.227.20:4450 , piece_id=C5F4WXTRZOBDRQM42HYC3TCDI2EB3KX625UBO57VYJJDKHHUIWGA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:16:17.269823246 (564h8m37.762255674s) [noise=true , address=145.79.227.27:4720 , piece_id=JPZ5PCHIKPVLDQOJ3UKKWLBY2KDT2EF6EZSSZUBULXELMDECABEA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: <nil> Sep 22 02:17:15.918034470 (564h9m36.410466903s) [noise=true , address=145.79.227.12:4787 , piece_id=IBVLLQFXHZZSML7RE5O6MRC44GBMPRUGOOXIW5VSW3DJUEIJOENA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1HiqDGVv9cDPmxCNevHz7HYV6Sna6KqHhgeBkys6zcrKZMXSyh, address:145.79.227.12:4787): protocol: rpc: dial tcp 145.79.227.12:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918254316 (564h9m36.410686753s) [noise=true , address=145.79.227.13:4836 , piece_id=2U4QXKGXUXPYCIYVI5BAMKQSYVP2REYVW2UMAZEUYZORI72C433A, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:145.79.227.13:4836): protocol: rpc: dial tcp 145.79.227.13:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918319211 (564h9m36.410751654s) [noise=true , address=145.79.227.16:4923 , piece_id=AXJ2HPZOGJHXJF52GB2LOX3EFQGSXEFJHK2YC66BKDHTDSZNBFUA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:145.79.227.16:4923): protocol: rpc: dial tcp 145.79.227.16:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918450129 (564h9m36.410882559s) [noise=true , address=145.79.227.5:4227 , piece_id=FOFYSH67V3LQIIBAEBGYDNKPNYPTPZQ6P2VB6XUBWAOTT3VJZKAQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:145.79.227.5:4227): protocol: rpc: dial tcp 145.79.227.5:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918489281 (564h9m36.41092171s) [noise=true , address=145.79.227.16:4924 , piece_id=VXKOIJMDQ5YHZCTNG46VGRYEPWS5D7BI7GZ24UGQRHKJUSZGDZQQ, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12mXBUcQEzsNgQ1YQzU9ezpZigBEHHhPttBac2G1Xrjks3SwM6z, address:145.79.227.16:4924): protocol: rpc: dial tcp 145.79.227.16:4924: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918527082 (564h9m36.410959511s) [noise=true , address=145.79.227.11:4753 , piece_id=LT3SZC6Z7P6MKGK7H6W5KAAOSM4V2IMPCAOLMM4XRKAIIUAVEFZA, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12RvrzrVY3N4FD2iNFBkHYBgitmrfcrLreLN6mJARGobdpbgtGu, address:145.79.227.11:4753): protocol: rpc: dial tcp 145.79.227.11:4753: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918538378 (564h9m36.410970823s) [noise=true , address=145.79.227.7:4324 , piece_id=TYC5E2IKQVW4AA5M5I34RNCZ5R6IHFEPALOZKHV2MJV3JZ3V2O2Q, seg_pos={0 0}, upload=496292]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:145.79.227.7:4324): protocol: rpc: dial tcp 145.79.227.7:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:15.918575562 (564h9m36.411008016s) [seg_pos={0 0}, upload=496292]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 02:17:16.071853105 (564h9m36.564285546s) [upload=496292]: Done uploading segments... Sep 22 02:17:16.071889485 (564h9m36.564321933s) [upload=496292]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 2.656035671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............................................XXXXXXX..X.X...] duration: 2.317318466s FAIL (2025-09-22 02:19:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 02:17:27.278111860 (564h9m47.770544312s) [upload=496310]: Starting upload Sep 22 02:17:27.278319587 (564h9m47.770752032s) [upload=496310]: Done starting upload Sep 22 02:17:27.278399633 (564h9m47.770832069s) [upload=496310]: Uploading segments... Sep 22 02:17:27.278679186 (564h9m47.771111629s) [upload=496310]: Got next segment. Inline: false Sep 22 02:17:27.278716987 (564h9m47.771149429s) [upload=496310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:17:27.278723741 (564h9m47.771156183s) [upload=496310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:17:27.391606475 (564h9m47.88403892s) [seg_pos={0 0}, upload=496310]: Begin upload segment... Sep 22 02:17:27.391656507 (564h9m47.884088952s) [seg_pos={0 0}, upload=496310]: Done begin upload segment. Sep 22 02:17:27.391684205 (564h9m47.884116647s) [noise=true , address=145.79.227.18:4006 , piece_id=LG4XNGKQYGDXAAT3N5326M2LZNJIF4567R3YDVRVTUEEZ3AXGPAQ, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.391694105 (564h9m47.884126551s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XJW3INL5BOB2GML6IXP7UVBSUH5B5ZPYJOMRNA2SJCT322QWTA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392068435 (564h9m47.884500876s) [noise=true , address=145.79.227.14:4840 , piece_id=7OV75GGOIRGJJ3UJ6KA2625HLQM5O7FYYIVKXZCKWWOBJXKP6J2A, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392079800 (564h9m47.884512242s) [noise=true , address=45.91.116.14:4191 , piece_id=BWEGJXAOKTPUIUZTPYIAJOF6IZASOSLHDVCYO4E475HB7RB2JQXQ, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392359813 (564h9m47.884792255s) [noise=true , address=145.79.227.4:4186 , piece_id=X2S2O7G2TO7W4NNTCHOUGTACSVAYPSJH4NNFXVNNXEUXQJSTGB6A, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392866275 (564h9m47.885298718s) [noise=true , address=45.91.116.13:4155 , piece_id=OKVAXFDLQDEXO4UBC4RFX44FIIKDHM6OSEL5QOV3ISQAOUJ5RSFA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.393199150 (564h9m47.885631585s) [noise=true , address=45.91.116.9:4261 , piece_id=BTQ6KKVFJWU732FB4YFSNO6HEINBXWAHVSU6FXPQHMSCXBWATZHA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.394785094 (564h9m47.887217537s) [noise=true , address=77.90.31.16:4734 , piece_id=24UDK3VHFEE242TEPT4OHFU26TW3PQ7C4NY5WGYRTDV7T37A22EA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.395727720 (564h9m47.888160168s) [noise=true , address=77.90.31.19:4843 , piece_id=O565PVZICSRTDI7DG3XVUM5REFUOXNEJT5G23UFFKWTQ5HBGRSLA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.396064317 (564h9m47.88849676s) [noise=true , address=145.79.227.15:4876 , piece_id=CATK7VYS34V6FM2TDA6WP6ZSCVL42WCRJQGMKMQE6B3KL2RUX5GA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.397050510 (564h9m47.889482958s) [noise=true , address=145.79.227.19:4079 , piece_id=GOC7R5364ANS74BUMZIKFYOQE5IOF4WIYYMBSQO7L5LFKH7T6MLA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.398371111 (564h9m47.890803571s) [noise=true , address=145.79.227.16:4912 , piece_id=WAWYX2DPWKMT2DWUQPLE3B5QR277HOYQJKAJ5RIPNQGY62V3FU2A, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.404113314 (564h9m47.896545774s) [noise=true , address=45.91.116.4:4937 , piece_id=PTEXK44Q5HARZBHVNXMIJQUYTYOEVR6JG7KOJTJTHX27AVO3V3IA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.404596127 (564h9m47.897028571s) [noise=true , address=145.79.227.16:4911 , piece_id=JCHWX24SAGWM67MQXDFY42FK7YXVREHIM4JBIXBKDZCQDHAPUTOA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.404928472 (564h9m47.897360901s) [noise=true , address=45.91.116.7:4010 , piece_id=SVNDPUPAZFAZH6ECKRTKZHTWO2CWBDFMC3ZHVA7YATN7EGAJYQYA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.411543951 (564h9m47.90397638s) [noise=true , address=45.91.116.13:4153 , piece_id=ZHHJ7HWJ3BZ7SOELVG5PKV7BECDBC6OTUPPWMX2IOKWOBFY5RIZA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.994508946 (564h9m48.486941392s) [upload=496310]: Next returned nil segment Sep 22 02:17:27.994529653 (564h9m48.4869621s) [upload=496310]: Waiting for error group managing segments... Sep 22 02:17:28.133038661 (564h9m48.625471091s) [noise=true , address=45.91.116.9:4261 , piece_id=BTQ6KKVFJWU732FB4YFSNO6HEINBXWAHVSU6FXPQHMSCXBWATZHA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.134870648 (564h9m48.627303084s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XJW3INL5BOB2GML6IXP7UVBSUH5B5ZPYJOMRNA2SJCT322QWTA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.136919100 (564h9m48.629351545s) [noise=true , address=45.91.116.13:4155 , piece_id=OKVAXFDLQDEXO4UBC4RFX44FIIKDHM6OSEL5QOV3ISQAOUJ5RSFA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.159786394 (564h9m48.65221883s) [noise=true , address=77.90.31.19:4843 , piece_id=O565PVZICSRTDI7DG3XVUM5REFUOXNEJT5G23UFFKWTQ5HBGRSLA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.161298102 (564h9m48.653730546s) [noise=true , address=45.91.116.4:4937 , piece_id=PTEXK44Q5HARZBHVNXMIJQUYTYOEVR6JG7KOJTJTHX27AVO3V3IA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.165445218 (564h9m48.65787766s) [noise=true , address=45.91.116.13:4153 , piece_id=ZHHJ7HWJ3BZ7SOELVG5PKV7BECDBC6OTUPPWMX2IOKWOBFY5RIZA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.170034775 (564h9m48.662467211s) [noise=true , address=77.90.31.16:4734 , piece_id=24UDK3VHFEE242TEPT4OHFU26TW3PQ7C4NY5WGYRTDV7T37A22EA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.173136107 (564h9m48.665568551s) [noise=true , address=45.91.116.7:4010 , piece_id=SVNDPUPAZFAZH6ECKRTKZHTWO2CWBDFMC3ZHVA7YATN7EGAJYQYA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.176492195 (564h9m48.668924642s) [noise=true , address=45.91.116.14:4191 , piece_id=BWEGJXAOKTPUIUZTPYIAJOF6IZASOSLHDVCYO4E475HB7RB2JQXQ, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:56.985562546 (564h10m17.477994998s) [noise=true , address=145.79.227.19:4079 , piece_id=GOC7R5364ANS74BUMZIKFYOQE5IOF4WIYYMBSQO7L5LFKH7T6MLA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:56.985662732 (564h10m17.478095164s) [noise=true , address=145.79.227.16:4912 , piece_id=WAWYX2DPWKMT2DWUQPLE3B5QR277HOYQJKAJ5RIPNQGY62V3FU2A, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985716627 (564h10m17.478149059s) [noise=true , address=145.79.227.4:4186 , piece_id=X2S2O7G2TO7W4NNTCHOUGTACSVAYPSJH4NNFXVNNXEUXQJSTGB6A, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985769530 (564h10m17.478201972s) [noise=true , address=145.79.227.15:4876 , piece_id=CATK7VYS34V6FM2TDA6WP6ZSCVL42WCRJQGMKMQE6B3KL2RUX5GA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985812353 (564h10m17.478244796s) [noise=true , address=145.79.227.14:4840 , piece_id=7OV75GGOIRGJJ3UJ6KA2625HLQM5O7FYYIVKXZCKWWOBJXKP6J2A, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985884301 (564h10m17.478316742s) [noise=true , address=145.79.227.16:4911 , piece_id=JCHWX24SAGWM67MQXDFY42FK7YXVREHIM4JBIXBKDZCQDHAPUTOA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: rpc: dial tcp 145.79.227.16:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985979172 (564h10m17.478411614s) [noise=true , address=145.79.227.18:4006 , piece_id=LG4XNGKQYGDXAAT3N5326M2LZNJIF4567R3YDVRVTUEEZ3AXGPAQ, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.986000804 (564h10m17.478433247s) [seg_pos={0 0}, upload=496310]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 02:17:57.096130906 (564h10m17.588563349s) [upload=496310]: Done uploading segments... Sep 22 02:17:57.096161057 (564h10m17.5885935s) [upload=496310]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.317373011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............................................XXXXXXX..X.X..] duration: 2.730317714s FAIL (2025-09-22 02:19:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 02:17:27.278111860 (564h9m47.770544312s) [upload=496310]: Starting upload Sep 22 02:17:27.278319587 (564h9m47.770752032s) [upload=496310]: Done starting upload Sep 22 02:17:27.278399633 (564h9m47.770832069s) [upload=496310]: Uploading segments... Sep 22 02:17:27.278679186 (564h9m47.771111629s) [upload=496310]: Got next segment. Inline: false Sep 22 02:17:27.278716987 (564h9m47.771149429s) [upload=496310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:17:27.278723741 (564h9m47.771156183s) [upload=496310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:17:27.391606475 (564h9m47.88403892s) [seg_pos={0 0}, upload=496310]: Begin upload segment... Sep 22 02:17:27.391656507 (564h9m47.884088952s) [seg_pos={0 0}, upload=496310]: Done begin upload segment. Sep 22 02:17:27.391684205 (564h9m47.884116647s) [noise=true , address=145.79.227.18:4006 , piece_id=LG4XNGKQYGDXAAT3N5326M2LZNJIF4567R3YDVRVTUEEZ3AXGPAQ, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.391694105 (564h9m47.884126551s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XJW3INL5BOB2GML6IXP7UVBSUH5B5ZPYJOMRNA2SJCT322QWTA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392068435 (564h9m47.884500876s) [noise=true , address=145.79.227.14:4840 , piece_id=7OV75GGOIRGJJ3UJ6KA2625HLQM5O7FYYIVKXZCKWWOBJXKP6J2A, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392079800 (564h9m47.884512242s) [noise=true , address=45.91.116.14:4191 , piece_id=BWEGJXAOKTPUIUZTPYIAJOF6IZASOSLHDVCYO4E475HB7RB2JQXQ, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392359813 (564h9m47.884792255s) [noise=true , address=145.79.227.4:4186 , piece_id=X2S2O7G2TO7W4NNTCHOUGTACSVAYPSJH4NNFXVNNXEUXQJSTGB6A, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392866275 (564h9m47.885298718s) [noise=true , address=45.91.116.13:4155 , piece_id=OKVAXFDLQDEXO4UBC4RFX44FIIKDHM6OSEL5QOV3ISQAOUJ5RSFA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.393199150 (564h9m47.885631585s) [noise=true , address=45.91.116.9:4261 , piece_id=BTQ6KKVFJWU732FB4YFSNO6HEINBXWAHVSU6FXPQHMSCXBWATZHA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.394785094 (564h9m47.887217537s) [noise=true , address=77.90.31.16:4734 , piece_id=24UDK3VHFEE242TEPT4OHFU26TW3PQ7C4NY5WGYRTDV7T37A22EA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.395727720 (564h9m47.888160168s) [noise=true , address=77.90.31.19:4843 , piece_id=O565PVZICSRTDI7DG3XVUM5REFUOXNEJT5G23UFFKWTQ5HBGRSLA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.396064317 (564h9m47.88849676s) [noise=true , address=145.79.227.15:4876 , piece_id=CATK7VYS34V6FM2TDA6WP6ZSCVL42WCRJQGMKMQE6B3KL2RUX5GA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.397050510 (564h9m47.889482958s) [noise=true , address=145.79.227.19:4079 , piece_id=GOC7R5364ANS74BUMZIKFYOQE5IOF4WIYYMBSQO7L5LFKH7T6MLA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.398371111 (564h9m47.890803571s) [noise=true , address=145.79.227.16:4912 , piece_id=WAWYX2DPWKMT2DWUQPLE3B5QR277HOYQJKAJ5RIPNQGY62V3FU2A, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.404113314 (564h9m47.896545774s) [noise=true , address=45.91.116.4:4937 , piece_id=PTEXK44Q5HARZBHVNXMIJQUYTYOEVR6JG7KOJTJTHX27AVO3V3IA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.404596127 (564h9m47.897028571s) [noise=true , address=145.79.227.16:4911 , piece_id=JCHWX24SAGWM67MQXDFY42FK7YXVREHIM4JBIXBKDZCQDHAPUTOA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.404928472 (564h9m47.897360901s) [noise=true , address=45.91.116.7:4010 , piece_id=SVNDPUPAZFAZH6ECKRTKZHTWO2CWBDFMC3ZHVA7YATN7EGAJYQYA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.411543951 (564h9m47.90397638s) [noise=true , address=45.91.116.13:4153 , piece_id=ZHHJ7HWJ3BZ7SOELVG5PKV7BECDBC6OTUPPWMX2IOKWOBFY5RIZA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.994508946 (564h9m48.486941392s) [upload=496310]: Next returned nil segment Sep 22 02:17:27.994529653 (564h9m48.4869621s) [upload=496310]: Waiting for error group managing segments... Sep 22 02:17:28.133038661 (564h9m48.625471091s) [noise=true , address=45.91.116.9:4261 , piece_id=BTQ6KKVFJWU732FB4YFSNO6HEINBXWAHVSU6FXPQHMSCXBWATZHA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.134870648 (564h9m48.627303084s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XJW3INL5BOB2GML6IXP7UVBSUH5B5ZPYJOMRNA2SJCT322QWTA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.136919100 (564h9m48.629351545s) [noise=true , address=45.91.116.13:4155 , piece_id=OKVAXFDLQDEXO4UBC4RFX44FIIKDHM6OSEL5QOV3ISQAOUJ5RSFA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.159786394 (564h9m48.65221883s) [noise=true , address=77.90.31.19:4843 , piece_id=O565PVZICSRTDI7DG3XVUM5REFUOXNEJT5G23UFFKWTQ5HBGRSLA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.161298102 (564h9m48.653730546s) [noise=true , address=45.91.116.4:4937 , piece_id=PTEXK44Q5HARZBHVNXMIJQUYTYOEVR6JG7KOJTJTHX27AVO3V3IA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.165445218 (564h9m48.65787766s) [noise=true , address=45.91.116.13:4153 , piece_id=ZHHJ7HWJ3BZ7SOELVG5PKV7BECDBC6OTUPPWMX2IOKWOBFY5RIZA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.170034775 (564h9m48.662467211s) [noise=true , address=77.90.31.16:4734 , piece_id=24UDK3VHFEE242TEPT4OHFU26TW3PQ7C4NY5WGYRTDV7T37A22EA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.173136107 (564h9m48.665568551s) [noise=true , address=45.91.116.7:4010 , piece_id=SVNDPUPAZFAZH6ECKRTKZHTWO2CWBDFMC3ZHVA7YATN7EGAJYQYA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.176492195 (564h9m48.668924642s) [noise=true , address=45.91.116.14:4191 , piece_id=BWEGJXAOKTPUIUZTPYIAJOF6IZASOSLHDVCYO4E475HB7RB2JQXQ, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:56.985562546 (564h10m17.477994998s) [noise=true , address=145.79.227.19:4079 , piece_id=GOC7R5364ANS74BUMZIKFYOQE5IOF4WIYYMBSQO7L5LFKH7T6MLA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:56.985662732 (564h10m17.478095164s) [noise=true , address=145.79.227.16:4912 , piece_id=WAWYX2DPWKMT2DWUQPLE3B5QR277HOYQJKAJ5RIPNQGY62V3FU2A, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985716627 (564h10m17.478149059s) [noise=true , address=145.79.227.4:4186 , piece_id=X2S2O7G2TO7W4NNTCHOUGTACSVAYPSJH4NNFXVNNXEUXQJSTGB6A, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985769530 (564h10m17.478201972s) [noise=true , address=145.79.227.15:4876 , piece_id=CATK7VYS34V6FM2TDA6WP6ZSCVL42WCRJQGMKMQE6B3KL2RUX5GA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985812353 (564h10m17.478244796s) [noise=true , address=145.79.227.14:4840 , piece_id=7OV75GGOIRGJJ3UJ6KA2625HLQM5O7FYYIVKXZCKWWOBJXKP6J2A, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985884301 (564h10m17.478316742s) [noise=true , address=145.79.227.16:4911 , piece_id=JCHWX24SAGWM67MQXDFY42FK7YXVREHIM4JBIXBKDZCQDHAPUTOA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: rpc: dial tcp 145.79.227.16:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985979172 (564h10m17.478411614s) [noise=true , address=145.79.227.18:4006 , piece_id=LG4XNGKQYGDXAAT3N5326M2LZNJIF4567R3YDVRVTUEEZ3AXGPAQ, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.986000804 (564h10m17.478433247s) [seg_pos={0 0}, upload=496310]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 02:17:57.096130906 (564h10m17.588563349s) [upload=496310]: Done uploading segments... Sep 22 02:17:57.096161057 (564h10m17.5885935s) [upload=496310]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.730354694s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 2.853645121s FAIL (2025-09-22 02:19:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.853668218s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.000842853s FAIL (2025-09-22 02:18:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000865028s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............................................XXXXXXX..X.X.] duration: 2.646122367s FAIL (2025-09-22 02:18:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 22 02:17:27.278111860 (564h9m47.770544312s) [upload=496310]: Starting upload Sep 22 02:17:27.278319587 (564h9m47.770752032s) [upload=496310]: Done starting upload Sep 22 02:17:27.278399633 (564h9m47.770832069s) [upload=496310]: Uploading segments... Sep 22 02:17:27.278679186 (564h9m47.771111629s) [upload=496310]: Got next segment. Inline: false Sep 22 02:17:27.278716987 (564h9m47.771149429s) [upload=496310]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 02:17:27.278723741 (564h9m47.771156183s) [upload=496310]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 02:17:27.391606475 (564h9m47.88403892s) [seg_pos={0 0}, upload=496310]: Begin upload segment... Sep 22 02:17:27.391656507 (564h9m47.884088952s) [seg_pos={0 0}, upload=496310]: Done begin upload segment. Sep 22 02:17:27.391684205 (564h9m47.884116647s) [noise=true , address=145.79.227.18:4006 , piece_id=LG4XNGKQYGDXAAT3N5326M2LZNJIF4567R3YDVRVTUEEZ3AXGPAQ, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.391694105 (564h9m47.884126551s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XJW3INL5BOB2GML6IXP7UVBSUH5B5ZPYJOMRNA2SJCT322QWTA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392068435 (564h9m47.884500876s) [noise=true , address=145.79.227.14:4840 , piece_id=7OV75GGOIRGJJ3UJ6KA2625HLQM5O7FYYIVKXZCKWWOBJXKP6J2A, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392079800 (564h9m47.884512242s) [noise=true , address=45.91.116.14:4191 , piece_id=BWEGJXAOKTPUIUZTPYIAJOF6IZASOSLHDVCYO4E475HB7RB2JQXQ, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392359813 (564h9m47.884792255s) [noise=true , address=145.79.227.4:4186 , piece_id=X2S2O7G2TO7W4NNTCHOUGTACSVAYPSJH4NNFXVNNXEUXQJSTGB6A, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.392866275 (564h9m47.885298718s) [noise=true , address=45.91.116.13:4155 , piece_id=OKVAXFDLQDEXO4UBC4RFX44FIIKDHM6OSEL5QOV3ISQAOUJ5RSFA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.393199150 (564h9m47.885631585s) [noise=true , address=45.91.116.9:4261 , piece_id=BTQ6KKVFJWU732FB4YFSNO6HEINBXWAHVSU6FXPQHMSCXBWATZHA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.394785094 (564h9m47.887217537s) [noise=true , address=77.90.31.16:4734 , piece_id=24UDK3VHFEE242TEPT4OHFU26TW3PQ7C4NY5WGYRTDV7T37A22EA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.395727720 (564h9m47.888160168s) [noise=true , address=77.90.31.19:4843 , piece_id=O565PVZICSRTDI7DG3XVUM5REFUOXNEJT5G23UFFKWTQ5HBGRSLA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.396064317 (564h9m47.88849676s) [noise=true , address=145.79.227.15:4876 , piece_id=CATK7VYS34V6FM2TDA6WP6ZSCVL42WCRJQGMKMQE6B3KL2RUX5GA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.397050510 (564h9m47.889482958s) [noise=true , address=145.79.227.19:4079 , piece_id=GOC7R5364ANS74BUMZIKFYOQE5IOF4WIYYMBSQO7L5LFKH7T6MLA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.398371111 (564h9m47.890803571s) [noise=true , address=145.79.227.16:4912 , piece_id=WAWYX2DPWKMT2DWUQPLE3B5QR277HOYQJKAJ5RIPNQGY62V3FU2A, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.404113314 (564h9m47.896545774s) [noise=true , address=45.91.116.4:4937 , piece_id=PTEXK44Q5HARZBHVNXMIJQUYTYOEVR6JG7KOJTJTHX27AVO3V3IA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.404596127 (564h9m47.897028571s) [noise=true , address=145.79.227.16:4911 , piece_id=JCHWX24SAGWM67MQXDFY42FK7YXVREHIM4JBIXBKDZCQDHAPUTOA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.404928472 (564h9m47.897360901s) [noise=true , address=45.91.116.7:4010 , piece_id=SVNDPUPAZFAZH6ECKRTKZHTWO2CWBDFMC3ZHVA7YATN7EGAJYQYA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.411543951 (564h9m47.90397638s) [noise=true , address=45.91.116.13:4153 , piece_id=ZHHJ7HWJ3BZ7SOELVG5PKV7BECDBC6OTUPPWMX2IOKWOBFY5RIZA, seg_pos={0 0}, upload=496310]: Uploading piece... Sep 22 02:17:27.994508946 (564h9m48.486941392s) [upload=496310]: Next returned nil segment Sep 22 02:17:27.994529653 (564h9m48.4869621s) [upload=496310]: Waiting for error group managing segments... Sep 22 02:17:28.133038661 (564h9m48.625471091s) [noise=true , address=45.91.116.9:4261 , piece_id=BTQ6KKVFJWU732FB4YFSNO6HEINBXWAHVSU6FXPQHMSCXBWATZHA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.134870648 (564h9m48.627303084s) [noise=true , address=45.91.116.6:4973 , piece_id=Q7XJW3INL5BOB2GML6IXP7UVBSUH5B5ZPYJOMRNA2SJCT322QWTA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.136919100 (564h9m48.629351545s) [noise=true , address=45.91.116.13:4155 , piece_id=OKVAXFDLQDEXO4UBC4RFX44FIIKDHM6OSEL5QOV3ISQAOUJ5RSFA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.159786394 (564h9m48.65221883s) [noise=true , address=77.90.31.19:4843 , piece_id=O565PVZICSRTDI7DG3XVUM5REFUOXNEJT5G23UFFKWTQ5HBGRSLA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.161298102 (564h9m48.653730546s) [noise=true , address=45.91.116.4:4937 , piece_id=PTEXK44Q5HARZBHVNXMIJQUYTYOEVR6JG7KOJTJTHX27AVO3V3IA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.165445218 (564h9m48.65787766s) [noise=true , address=45.91.116.13:4153 , piece_id=ZHHJ7HWJ3BZ7SOELVG5PKV7BECDBC6OTUPPWMX2IOKWOBFY5RIZA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.170034775 (564h9m48.662467211s) [noise=true , address=77.90.31.16:4734 , piece_id=24UDK3VHFEE242TEPT4OHFU26TW3PQ7C4NY5WGYRTDV7T37A22EA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.173136107 (564h9m48.665568551s) [noise=true , address=45.91.116.7:4010 , piece_id=SVNDPUPAZFAZH6ECKRTKZHTWO2CWBDFMC3ZHVA7YATN7EGAJYQYA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:28.176492195 (564h9m48.668924642s) [noise=true , address=45.91.116.14:4191 , piece_id=BWEGJXAOKTPUIUZTPYIAJOF6IZASOSLHDVCYO4E475HB7RB2JQXQ, seg_pos={0 0}, upload=496310]: Done uploading piece. err: <nil> Sep 22 02:17:56.985562546 (564h10m17.477994998s) [noise=true , address=145.79.227.19:4079 , piece_id=GOC7R5364ANS74BUMZIKFYOQE5IOF4WIYYMBSQO7L5LFKH7T6MLA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: context deadline exceeded; context deadline exceeded Sep 22 02:17:56.985662732 (564h10m17.478095164s) [noise=true , address=145.79.227.16:4912 , piece_id=WAWYX2DPWKMT2DWUQPLE3B5QR277HOYQJKAJ5RIPNQGY62V3FU2A, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985716627 (564h10m17.478149059s) [noise=true , address=145.79.227.4:4186 , piece_id=X2S2O7G2TO7W4NNTCHOUGTACSVAYPSJH4NNFXVNNXEUXQJSTGB6A, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: rpc: dial tcp 145.79.227.4:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985769530 (564h10m17.478201972s) [noise=true , address=145.79.227.15:4876 , piece_id=CATK7VYS34V6FM2TDA6WP6ZSCVL42WCRJQGMKMQE6B3KL2RUX5GA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: rpc: dial tcp 145.79.227.15:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985812353 (564h10m17.478244796s) [noise=true , address=145.79.227.14:4840 , piece_id=7OV75GGOIRGJJ3UJ6KA2625HLQM5O7FYYIVKXZCKWWOBJXKP6J2A, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985884301 (564h10m17.478316742s) [noise=true , address=145.79.227.16:4911 , piece_id=JCHWX24SAGWM67MQXDFY42FK7YXVREHIM4JBIXBKDZCQDHAPUTOA, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: rpc: dial tcp 145.79.227.16:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.985979172 (564h10m17.478411614s) [noise=true , address=145.79.227.18:4006 , piece_id=LG4XNGKQYGDXAAT3N5326M2LZNJIF4567R3YDVRVTUEEZ3AXGPAQ, seg_pos={0 0}, upload=496310]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: rpc: dial tcp 145.79.227.18:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 02:17:56.986000804 (564h10m17.478433247s) [seg_pos={0 0}, upload=496310]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 22 02:17:57.096130906 (564h10m17.588563349s) [upload=496310]: Done uploading segments... Sep 22 02:17:57.096161057 (564h10m17.5885935s) [upload=496310]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.646218116s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)