Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.354564688s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:50:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.354653667s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.299474058s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:49:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.299505277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.349169281s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:48:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.349189616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.402871679s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:47:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.402959987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.31684518s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:46:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.31686378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.359586101s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:46:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.359618502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.380348884s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:45:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.380373231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.410949606s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:44:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.41097411s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.458947622s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:43:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.4589783s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.492220274s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:43:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.492265704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.358853933s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:42:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.358879405s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.374970604s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:41:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.37499393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.366568214s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:41:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.366599417s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.326075336s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:39:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.326101094s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.308908688s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:38:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.308997632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.434969339s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:38:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.435109289s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.335860915s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:37:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.335888649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.397567196s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:36:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397598751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.387656267s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:36:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.387684398s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.273110479s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:35:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.273220901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.334072847s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:34:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.33415808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.308785496s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:33:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.308818544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.412203651s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:32:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.412229654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.413787826s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:31:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.413813871s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.474798914s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:31:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.474831505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.27436738s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:30:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.274450546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.369695112s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:29:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.369721279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.428770186s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:28:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.428860515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.427412274s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:28:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.427439148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.410536477s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.410561714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.319718331s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:28:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.319798899s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.432835377s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:26:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.432865191s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.545164539s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:26:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.545195377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.311755518s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:25:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.311783769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.286856309s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:24:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.286884743s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.371190853s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:24:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.371214581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.367655559s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:23:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.367685507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.272210752s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:22:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.272353724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.326807675s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:21:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.326850959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.292779038s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:21:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.292800332s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.400056343s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:20:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.400167079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.269644421s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:19:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.269671416s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.454136141s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:18:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.454163891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.35565098s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:17:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.355681897s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.418037457s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:16:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.418068936s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.583173584s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:15:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.583216929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.344334392s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:14:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.344359237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.407486814s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:14:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.407508326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.397610988s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:12:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397635906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.379170709s trace: https://jaeger.dev.storj.io/trace/f7d532537c6a177 FAIL (2025-12-03 15:12:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments... Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment... Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment. Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece... Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments... Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil> Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments... Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.379193035s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)