Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.46425768s trace:
https://jaeger.dev.storj.io/trace/f7d532537c6a177
FAIL (2025-12-03 15:11:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload
Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload
Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments...
Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false
Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment...
Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment.
Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment
Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments...
Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF
Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments...
Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.464290544s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.491051016s trace:
https://jaeger.dev.storj.io/trace/f7d532537c6a177
FAIL (2025-12-03 15:11:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload
Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload
Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments...
Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false
Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment...
Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment.
Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment
Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments...
Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF
Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments...
Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.491084043s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.43532395s trace:
https://jaeger.dev.storj.io/trace/f7d532537c6a177
FAIL (2025-12-03 15:10:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload
Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload
Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments...
Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false
Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment...
Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment.
Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment
Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments...
Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF
Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments...
Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.435354509s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.11938706s trace:
https://jaeger.dev.storj.io/trace/f7d532537c6a177
FAIL (2025-12-03 15:09:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 3 15:08:52.896944447 (533h24m38.255603091s) [upload=469737]: Starting upload
Dec 3 15:08:52.897063953 (533h24m38.255722618s) [upload=469737]: Done starting upload
Dec 3 15:08:52.897163121 (533h24m38.255821763s) [upload=469737]: Uploading segments...
Dec 3 15:08:52.897558071 (533h24m38.256216729s) [upload=469737]: Got next segment. Inline: false
Dec 3 15:08:52.897640308 (533h24m38.256298963s) [upload=469737]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 15:08:52.897647833 (533h24m38.256306492s) [upload=469737]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 15:08:53.009235713 (533h24m38.36789437s) [seg_pos={0 0}, upload=469737]: Begin upload segment...
Dec 3 15:08:53.009288243 (533h24m38.367946901s) [seg_pos={0 0}, upload=469737]: Done begin upload segment.
Dec 3 15:08:53.009367478 (533h24m38.368026138s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.009421856 (533h24m38.368080517s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010027565 (533h24m38.368686223s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010043456 (533h24m38.36870212s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010102225 (533h24m38.368760878s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010428865 (533h24m38.369087523s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010654013 (533h24m38.369312666s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.010809455 (533h24m38.369468117s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011020194 (533h24m38.36967885s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011173200 (533h24m38.369832083s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.011370267 (533h24m38.37002892s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.012153095 (533h24m38.370811749s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.012620564 (533h24m38.371279222s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.013017089 (533h24m38.371675751s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.013429742 (533h24m38.372088398s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.014085224 (533h24m38.372743864s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Uploading piece...
Dec 3 15:08:53.383294268 (533h24m38.741952924s) [noise=true , address=77.90.31.12:4589 , piece_id=QYLEZMT2J7G55HPDIPCFEL3RHR32OD52WR7VU3DVJWRJXK6BUCTA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.391425714 (533h24m38.750084376s) [noise=true , address=77.90.31.13:4628 , piece_id=Y3YJUINEXGZAJ77P62XUWHLUJHGMAYOT4KOP74KCAMRTURHFLHAA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.392199080 (533h24m38.750857733s) [noise=true , address=77.90.31.19:4841 , piece_id=TDKX6QY6TTTSKMXSB26GUOXUDRJL7JWHGMWH733GWR5LO5UG7OKA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 3 15:08:53.615165191 (533h24m38.973823844s) [upload=469737]: Next returned nil segment
Dec 3 15:08:53.615187725 (533h24m38.973846378s) [upload=469737]: Waiting for error group managing segments...
Dec 3 15:08:53.748126101 (533h24m39.106784759s) [noise=true , address=145.79.227.31:4799 , piece_id=RQISAF6FHRW3JUUE4YLF5IVXQEZ5A43BKYM4U57DA6IGLHD4SIBQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.751792521 (533h24m39.110451178s) [noise=true , address=145.79.227.40:4050 , piece_id=KANM5425LF7NBC2TJAFKYJ7AVDVXHH2MCVF4OSH5N362E3DVHNDA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.756246928 (533h24m39.114905591s) [noise=true , address=145.79.227.17:4971 , piece_id=TSDHR6Z3TWSKOI6IYIL3SEYL3W36DTWIMNMUFSI6WSMNW4P2W73A, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.757265849 (533h24m39.115924506s) [noise=true , address=145.79.227.32:4835 , piece_id=SIATHQQIG7C5ZTWTVVYHDZARADDU4B7USV36PP5ASLCLJOFHSFYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.761193601 (533h24m39.119852264s) [noise=true , address=145.79.227.30:4762 , piece_id=UGG32EM722WBUFDCBGM5IGL6MALYFVGUTGHRCSYAI42NG2RTLSYQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.764259910 (533h24m39.122918564s) [noise=true , address=77.90.31.14:4662 , piece_id=4S6OGEG3RXMGWZKOKDVDXYQWLYXZKGCRSLAVOAN3Q3DBI6OK3XIQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.766655094 (533h24m39.125313755s) [noise=true , address=145.79.227.25:4617 , piece_id=37OS4W3JINQZAM5MIYUZKQTAPIELOWEJNF6RANWDANCFZEA3U3NQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.790946297 (533h24m39.149604941s) [noise=true , address=77.90.31.10:4519 , piece_id=ZG5RCZBS5EMJPYP5AJPA5TQ4WTZTR3GGRLPQJVD5RTUQ5YVCFEQA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.793802746 (533h24m39.152461402s) [noise=true , address=77.90.31.17:4772 , piece_id=5FDMW3BAIOCJ6QD7H425AVHGF3M4M3FXOFR333F5QFIXOKDZGUCA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.834197204 (533h24m39.192855858s) [noise=true , address=77.90.31.13:4627 , piece_id=URXBZRB56WYW5MEFPE2VNYVLAYJXVZI7SRSU4XDA6YK7HIMLBZ2Q, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.851695423 (533h24m39.21035408s) [noise=true , address=77.90.31.21:4878 , piece_id=S5LHSQ4WW2M6IZ5EX5AETD6I3MZOEBIYBQVDNJPYMAPZQEPJGIRQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:08:53.864177884 (533h24m39.222836527s) [noise=true , address=145.79.227.33:4871 , piece_id=U2TGKL2AX6ELCI3KCWMUKW4WTTYAZAEMKWBX7JJUIYTXR5ZYUFTQ, seg_pos={0 0}, upload=469737]: Done uploading piece. err: <nil>
Dec 3 15:09:22.611920068 (533h25m7.970578728s) [noise=true , address=77.90.31.12:4590 , piece_id=CI5XUQ57ZSW77S6DOYCTAHQ5TTC4P2FNRIZSMHA5SYVZXZEEUIMA, seg_pos={0 0}, upload=469737]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: expected piece hash; context deadline exceeded; EOF
Dec 3 15:09:22.611954511 (533h25m7.970613151s) [seg_pos={0 0}, upload=469737]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 3 15:09:22.729429933 (533h25m8.088088595s) [upload=469737]: Done uploading segments...
Dec 3 15:09:22.729464912 (533h25m8.088123572s) [upload=469737]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.11940369s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
1.65455671s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:49:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.654593938s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
1.534288723s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:49:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.534314135s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.350888395s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:48:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.350905632s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
1.621646232s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:48:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.621672184s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.16597739s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:48:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.166005275s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [X..X........................................................] duration:
1.668412219s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:47:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.668437447s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.X..X.......................................................] duration:
1.6055452s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:47:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.605569576s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X.X.........................................................] duration:
2.328109568s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:47:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.328156843s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..X..X......................................................] duration:
1.650706677s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:46:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.650739199s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X.X........................................................] duration:
2.429908429s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:46:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.430024768s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [X..X..X.....................................................] duration:
1.644019238s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:45:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.644040918s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.X..X..X....................................................] duration:
1.612687799s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:45:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.612713139s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..X..X..X...................................................] duration:
1.598654639s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:44:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.598682306s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration:
2.221611798s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:44:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.221638534s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...X..X..X..................................................] duration:
1.568544607s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:43:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.568567046s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X.X......................................................] duration:
2.356959996s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:43:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.356994583s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [....X..X..X.................................................] duration:
1.581447533s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:43:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.581478005s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X.X.....................................................] duration:
2.4432948s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:42:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.443320015s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.....X..X..X................................................] duration:
1.639922456s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:42:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.639949272s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X.X....................................................] duration:
2.410430576s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:42:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.410456176s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X.X...................................................] duration:
2.282572955s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:41:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.282599832s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [......X..X..X...............................................] duration:
1.644066695s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:41:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.644127294s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X.X..................................................] duration:
2.253872321s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:41:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.253900433s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X.X.................................................] duration:
2.253441955s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:40:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.25348865s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.......X..X..X..............................................] duration:
2.022049352s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:40:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.022075711s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X.X................................................] duration:
2.29882371s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:40:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.298852775s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X.X...............................................] duration:
2.249435744s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:39:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.249458323s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [........X..X..X.............................................] duration:
1.664920721s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:39:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.664946709s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.........X..X..X............................................] duration:
1.595087126s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:39:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.595135755s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X.X..............................................] duration:
2.18320637s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:39:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.183234426s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..........X..X..X...........................................] duration:
1.675205034s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:38:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.675237979s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X.X.............................................] duration:
2.362850205s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:38:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.362873215s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...........X..X..X..........................................] duration:
1.64022948s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:37:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.640252872s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [............X..X..X.........................................] duration:
1.687441927s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:37:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.687466415s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X.X............................................] duration:
2.347106804s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:36:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.347133505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.............X..X..X........................................] duration:
1.686531575s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:36:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.686560802s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..............X..X..X.......................................] duration:
1.63452514s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:35:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.634555043s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X.X...........................................] duration:
2.317941444s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:35:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.317958985s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...............X..X..X......................................] duration:
1.7882769s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:35:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.788302255s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X.X..........................................] duration:
2.322759819s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:35:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.322797941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [................X..X..X.....................................] duration:
1.677203263s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:35:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.677224639s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X.X.........................................] duration:
2.462969518s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:35:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.46300637s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.................X..X..X....................................] duration:
1.653967688s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:34:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.653999506s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..................X..X..X...................................] duration:
1.606546198s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:34:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.606567302s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X.X........................................] duration:
2.259334258s trace:
https://jaeger.dev.storj.io/trace/3d647576540b8a43
FAIL (2025-12-03 03:34:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
9127]: Got next segment. Inline: false
Dec 3 03:05:10.312598337 (521h20m55.671256995s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 3 03:05:10.312608803 (521h20m55.671267492s) [upload=459127]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 3 03:05:10.423211631 (521h20m55.781870284s) [seg_pos={0 0}, upload=459127]: Begin upload segment...
Dec 3 03:05:10.423372378 (521h20m55.782031031s) [seg_pos={0 0}, upload=459127]: Done begin upload segment.
Dec 3 03:05:10.423418556 (521h20m55.782077218s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424211836 (521h20m55.782870498s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424340565 (521h20m55.782999228s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424885149 (521h20m55.783543809s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.424952814 (521h20m55.783611477s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425383646 (521h20m55.7840423s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425439223 (521h20m55.784097883s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.425842229 (521h20m55.784500887s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426000165 (521h20m55.784658824s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426313937 (521h20m55.7849726s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426609591 (521h20m55.78526825s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.426729300 (521h20m55.785387955s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427033135 (521h20m55.785691793s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427073886 (521h20m55.78573254s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.427456960 (521h20m55.786115614s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429209200 (521h20m55.787867864s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.429655599 (521h20m55.788314246s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.430923538 (521h20m55.789582195s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Uploading piece...
Dec 3 03:05:10.978972595 (521h20m56.337631249s) [upload=459127]: Next returned nil segment
Dec 3 03:05:10.978986284 (521h20m56.337644954s) [upload=459127]: Waiting for error group managing segments...
Dec 3 03:05:11.125651281 (521h20m56.484309925s) [noise=true , address=145.79.227.18:4025 , piece_id=QEP5Z34FDLH4QED6LVUMRF3X7OHJ5XEEAH4DJIBPZ2KSJWOE5QRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.126430562 (521h20m56.485089219s) [noise=true , address=77.90.31.15:4722 , piece_id=AQVLQVGFOO2OMNH5PBGPGKVS7XITJKIUDZWW5RLTXDRH73YD7U7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.128646531 (521h20m56.487305186s) [noise=true , address=145.79.227.37:4587 , piece_id=GAHPCJNKCEIT56NWJUDQGDDB6FFCB7FRTJLK6RRL4RZAV22QGFRA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.135664103 (521h20m56.494322767s) [noise=true , address=77.90.31.10:4521 , piece_id=D6UE43RSSQM6KXXIQOWNWQCMVRLDTBGOIBQ4RWYP7APKY2A7WZ6A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.148194132 (521h20m56.506852791s) [noise=true , address=77.90.31.12:4598 , piece_id=7QOESFFRIX5JQBVK7XJLX43Y52BDV534GG7S6PYSA6PLEYV2GV7Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.171572040 (521h20m56.530230731s) [noise=true , address=77.90.31.21:4882 , piece_id=LMNAVASZS5NK3XMMU5G4KICMFUZFJ4MJDT4ZM2SKNL32VB3I7EWQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.254222716 (521h20m56.612881371s) [noise=true , address=145.79.227.6:4268 , piece_id=GVQ3UPCNWCJNFNQ4567SPHSNIXC56YCMXDKUGCAHBI4ODEMMNRLQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.267623885 (521h20m56.626282544s) [noise=true , address=77.90.31.19:4853 , piece_id=PV2KOS52WKXCBOR3LD3BSSPRI3J4DRBL3Y56NMIS25YNAZ6564BA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.338767913 (521h20m56.697426567s) [noise=true , address=145.79.227.52:4461 , piece_id=XTIZVXAY7EOLDI4O75AUEPKECNU6SUPOEIDMECSLDJJQ6PSRLMOA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.556892691 (521h20m56.915551345s) [noise=true , address=145.79.227.15:4881 , piece_id=7XQQQ2RITUURKR4OEAUBLA4XELAOQGF2GBSPDTDJFCSVE2MMLU2A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:11.707227773 (521h20m57.065886436s) [noise=true , address=64.49.13.27:4959 , piece_id=ELBRWAMWPUGFQTIS5G4HFSU56H5X6LRF36MVAZ7KUM5K7TIBXCIQ, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:05:12.625208900 (521h20m57.983867573s) [noise=true , address=64.49.13.169:4716 , piece_id=RSP4YC77IXI7PVF2HDQE53XXSS77HKZ4UNHLAFOQDON653NTXUWA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:06.710288122 (521h21m52.068946822s) [noise=true , address=45.91.116.7:4027 , piece_id=BZFGQPBTMDWCLFF2IAKE3CPEP6RT77OVCLYWDH63AEMQ6ZYQROPA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: <nil>
Dec 3 03:06:10.034313264 (521h21m55.392971929s) [noise=true , address=45.91.116.9:4293 , piece_id=ABUJYELKK4MH2PIHBZZ4UGPU4NQ2L3TVBDT7XQFOPHV3BN4CHR2Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1XMXDwZP5R7ZZGr8x57g3faasnxk7YJiLeBSEodBeCP3mnPuRG, address:45.91.116.9:4293): protocol: rpc: dial tcp 45.91.116.9:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034403415 (521h21m55.39306208s) [noise=true , address=45.91.116.5:4316 , piece_id=ARSPQEVDIR6M3LXF7QEKWT5J6ZVAR7JVLRZEDOSLB3V7Y6TY3OFA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1viKEvjPFGZ9E6E9DFPNXZWYE6SYoh4gzxNYqGGK6Mt2q9YciQ, address:45.91.116.5:4316): protocol: rpc: dial tcp 45.91.116.5:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034525189 (521h21m55.393183851s) [noise=true , address=45.91.116.13:4184 , piece_id=UZEWGFXUCY2V2UEO2WLD7PZBAEZPBL4W4B2BCBIGO54AUVZKMWYA, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12dfL2FMGtGXzuLLnjpGLQ4nV8EKCdKnwVFBqzpLjehQ8AqjkCx, address:45.91.116.13:4184): protocol: read tcp 139.59.144.22:56990->45.91.116.13:4184: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034725076 (521h21m55.39338374s) [noise=true , address=45.91.116.13:4165 , piece_id=EI3HOM5UX553336EQ3ZZSHTV23326YHXAFQUDRNX2EAFZZOWX72Q, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:12XR8ACsXNSFvcZU3boexUojKn2Mr2i7fAbBC6aQ6imCH995HKg, address:45.91.116.13:4165): protocol: write tcp 139.59.144.22:38346->45.91.116.13:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.034977130 (521h21m55.39363579s) [noise=true , address=45.91.116.11:4149 , piece_id=PFOFO3DE6IF6I4UM6DKANQ3VX57SFFIJR7XSKPTKLU7WRJPENF7A, seg_pos={0 0}, upload=459127]: Done uploading piece. err: ecclient: upload failed (node:1mRhwD3dneKTkyiw9yNQPSr6AD1y1yyvZDAbCmB46siz7ywrnA, address:45.91.116.11:4149): protocol: read tcp 139.59.144.22:57658->45.91.116.11:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 3 03:06:10.035024932 (521h21m55.393683589s) [seg_pos={0 0}, upload=459127]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Dec 3 03:06:10.139157030 (521h21m55.497815689s) [upload=459127]: Done uploading segments...
Dec 3 03:06:10.139189317 (521h21m55.497847973s) [upload=459127]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 2.259359623s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...................X..X..X..................................] duration:
1.545561719s trace:
https://jaeger.dev.storj.io/trace/5e0397fa063b3702
FAIL (2025-12-03 03:33:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.545578394s
core
uplink
us-select-1
us1-placement-20
|