Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.................X..X...X..X..X........X..X..X...X..........] duration:
22.263986025s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:09:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 22.264013852s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [..................X..X...X..X..X........X..X..X...X.........] duration:
21.032399648s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:08:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 21.032430213s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [...................XXX.XXX...............X..................] duration:
20.116431138s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:08:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.116496317s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
43 out of 60 is failed (threshold=3) [................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration:
30.112113798s trace:
https://jaeger.dev.storj.io/trace/25e4b0e275e1c2a6
FAIL (2026-03-18 20:08:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Mar 18 20:08:04.675539613 (195h57m7.691209213s) [upload=188219]: Starting upload
Mar 18 20:08:04.675616393 (195h57m7.691285996s) [upload=188219]: Done starting upload
Mar 18 20:08:04.675680845 (195h57m7.691350435s) [upload=188219]: Uploading segments...
Mar 18 20:08:04.675945655 (195h57m7.691615251s) [upload=188219]: Got next segment. Inline: false
Mar 18 20:08:04.675999995 (195h57m7.691669586s) [upload=188219]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:08:04.676006689 (195h57m7.691676296s) [upload=188219]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:08:08.542180430 (195h57m11.557850049s) [seg_pos={0 0}, upload=188219]: Begin upload segment...
Mar 18 20:08:08.542296610 (195h57m11.557966226s) [seg_pos={0 0}, upload=188219]: Done begin upload segment.
Mar 18 20:08:08.542312966 (195h57m11.557982569s) [noise=true , address=77.90.31.13:4630 , piece_id=GNFFOJEQILYSRLQY32T4YSZ2OIGLDXRW6EFLS77XZTG4VAHSQNKQ, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.542377567 (195h57m11.558047184s) [noise=true , address=77.90.31.17:4770 , piece_id=QXOEI54K75JLVK6XMNXUM65Z2LSVXXPBQ4AC6TW42GQOHGWEOPZQ, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.542987814 (195h57m11.558657433s) [noise=true , address=45.91.116.4:4941 , piece_id=CTZGJRACETCYD7PFMXE22GHOCCZ5WOODDO2DVZ5OSFE5XPD6NK7A, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.543919192 (195h57m11.559588786s) [noise=true , address=145.79.227.24:4585 , piece_id=B2SQ24DTYDDNNFUTN7Z2TDPK7RBRONW74ZCYVA2FC3KQ2ZGLKVHQ, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.544245354 (195h57m11.559914957s) [noise=true , address=45.91.116.11:4118 , piece_id=LEQW2YI43ERPYL5JW6MTKJGGWXL2XV4DGVRDSIKHJRE26UDZ2FJA, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.544509094 (195h57m11.560178697s) [noise=true , address=45.91.116.9:4265 , piece_id=5VWNXSW65YGFWZ7ERAZGQBMT6LCRWMIBXEZS3PPG6ENASK7EA5VQ, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.544779681 (195h57m11.560449281s) [noise=true , address=77.90.31.18:4809 , piece_id=KVAABLT4S7IXL7D6YCDWMHUKFOY7CSJRQMVV445WDSPF45EUS2TQ, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.545053943 (195h57m11.560723547s) [noise=true , address=145.79.227.46:4601 , piece_id=LNKLL5XF2EPFEVZG7ORHAAB4YAFYX5YN2CTWVG23R25QQQYNCOFA, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.545355009 (195h57m11.56102461s) [noise=true , address=145.79.227.44:4193 , piece_id=C6NZ3FA6XEIPY6I72FSS3LWNHGXZJ7OLP3TCVBHFWU7TS4LNDUHQ, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.545880783 (195h57m11.561550382s) [noise=true , address=77.90.31.15:4700 , piece_id=C5BXI6M6L6L2SFAIHMNRRJBXYEZA6TZ7EVZACJ7RTXGOCRVDVLCA, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.546267032 (195h57m11.561936632s) [noise=true , address=45.91.116.10:4081 , piece_id=OVLOKDHC7AA2EFFOJXF2JO2ANCIXHEWVQAGG7SQBXZQWEG3WQLFA, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.546617309 (195h57m11.562286912s) [noise=true , address=79.127.219.46:4891 , piece_id=4HNXV6EG2FT443HXICKGZBBLH46X42SWLJEWGLJJCHCZS5HG5JSQ, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.547028929 (195h57m11.562698529s) [noise=true , address=145.79.227.47:4267 , piece_id=JBA5ETD6VGFLLF6WKWEISBH3RT2WVGIJFOW3QFUL2SSDRSG6HEFA, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.547415330 (195h57m11.56308495s) [noise=true , address=77.90.31.11:4558 , piece_id=CPP47TSSLJQGHJGZRK3FIOIGD2WEKJ7IE5CLF3RZQSH4EKB7MS3Q, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.547849807 (195h57m11.563519412s) [noise=true , address=79.127.219.41:4929 , piece_id=SFTW6PSMADO77ZL6T3W7TMBRMVIFKICSUX6FG2IEV43NEBRI73RQ, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:08.548247069 (195h57m11.563916672s) [noise=true , address=45.91.116.8:4047 , piece_id=DZYYIPGEJCU4TQ6S6ANDIBM575AVRVDOTSTVOH347ZI57VEMEBFQ, seg_pos={0 0}, upload=188219]: Uploading piece...
Mar 18 20:08:09.139847786 (195h57m12.155517396s) [upload=188219]: Next returned nil segment
Mar 18 20:08:09.139867174 (195h57m12.155536784s) [upload=188219]: Waiting for error group managing segments...
Mar 18 20:08:09.271204050 (195h57m12.286873659s) [noise=true , address=77.90.31.13:4630 , piece_id=GNFFOJEQILYSRLQY32T4YSZ2OIGLDXRW6EFLS77XZTG4VAHSQNKQ, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.289258453 (195h57m12.304928064s) [noise=true , address=77.90.31.18:4809 , piece_id=KVAABLT4S7IXL7D6YCDWMHUKFOY7CSJRQMVV445WDSPF45EUS2TQ, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.290986807 (195h57m12.306656414s) [noise=true , address=77.90.31.11:4558 , piece_id=CPP47TSSLJQGHJGZRK3FIOIGD2WEKJ7IE5CLF3RZQSH4EKB7MS3Q, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.291728668 (195h57m12.307398268s) [noise=true , address=45.91.116.9:4265 , piece_id=5VWNXSW65YGFWZ7ERAZGQBMT6LCRWMIBXEZS3PPG6ENASK7EA5VQ, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.292567515 (195h57m12.30823712s) [noise=true , address=45.91.116.10:4081 , piece_id=OVLOKDHC7AA2EFFOJXF2JO2ANCIXHEWVQAGG7SQBXZQWEG3WQLFA, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.297873475 (195h57m12.313543078s) [noise=true , address=45.91.116.4:4941 , piece_id=CTZGJRACETCYD7PFMXE22GHOCCZ5WOODDO2DVZ5OSFE5XPD6NK7A, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.298640040 (195h57m12.314309643s) [noise=true , address=45.91.116.8:4047 , piece_id=DZYYIPGEJCU4TQ6S6ANDIBM575AVRVDOTSTVOH347ZI57VEMEBFQ, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.298656744 (195h57m12.314326344s) [noise=true , address=45.91.116.11:4118 , piece_id=LEQW2YI43ERPYL5JW6MTKJGGWXL2XV4DGVRDSIKHJRE26UDZ2FJA, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.301376410 (195h57m12.317046015s) [noise=true , address=77.90.31.17:4770 , piece_id=QXOEI54K75JLVK6XMNXUM65Z2LSVXXPBQ4AC6TW42GQOHGWEOPZQ, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.310819861 (195h57m12.326489467s) [noise=true , address=77.90.31.15:4700 , piece_id=C5BXI6M6L6L2SFAIHMNRRJBXYEZA6TZ7EVZACJ7RTXGOCRVDVLCA, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.342322008 (195h57m12.357991615s) [noise=true , address=79.127.219.46:4891 , piece_id=4HNXV6EG2FT443HXICKGZBBLH46X42SWLJEWGLJJCHCZS5HG5JSQ, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:09.360220216 (195h57m12.375889817s) [noise=true , address=79.127.219.41:4929 , piece_id=SFTW6PSMADO77ZL6T3W7TMBRMVIFKICSUX6FG2IEV43NEBRI73RQ, seg_pos={0 0}, upload=188219]: Done uploading piece. err: <nil>
Mar 18 20:08:34.374166953 (195h57m37.389836562s) [noise=true , address=145.79.227.24:4585 , piece_id=B2SQ24DTYDDNNFUTN7Z2TDPK7RBRONW74ZCYVA2FC3KQ2ZGLKVHQ, seg_pos={0 0}, upload=188219]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: rpc: dial tcp 145.79.227.24:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:08:34.374397036 (195h57m37.390066645s) [noise=true , address=145.79.227.47:4267 , piece_id=JBA5ETD6VGFLLF6WKWEISBH3RT2WVGIJFOW3QFUL2SSDRSG6HEFA, seg_pos={0 0}, upload=188219]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: rpc: dial tcp 145.79.227.47:4267: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:08:34.374686619 (195h57m37.390356228s) [noise=true , address=145.79.227.46:4601 , piece_id=LNKLL5XF2EPFEVZG7ORHAAB4YAFYX5YN2CTWVG23R25QQQYNCOFA, seg_pos={0 0}, upload=188219]: Done uploading piece. err: ecclient: upload failed (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, address:145.79.227.46:4601): protocol: rpc: dial tcp 145.79.227.46:4601: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:08:34.374955193 (195h57m37.390624801s) [noise=true , address=145.79.227.44:4193 , piece_id=C6NZ3FA6XEIPY6I72FSS3LWNHGXZJ7OLP3TCVBHFWU7TS4LNDUHQ, seg_pos={0 0}, upload=188219]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: rpc: dial tcp 145.79.227.44:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:08:34.374993756 (195h57m37.390663359s) [seg_pos={0 0}, upload=188219]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Mar 18 20:08:34.484509842 (195h57m37.500179435s) [upload=188219]: Done uploading segments...
Mar 18 20:08:34.484547149 (195h57m37.500216749s) [upload=188219]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.112156269s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [.............XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001577643s trace:
https://jaeger.dev.storj.io/trace/5d7574cdf2397975
FAIL (2026-03-18 20:08:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00160442s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [...................X..X...X..X..X........X..X..X...X........] duration:
18.315374226s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:08:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 18.315402774s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
42 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration:
30.113373814s trace:
https://jaeger.dev.storj.io/trace/77ad14034354a65b
FAIL (2026-03-18 20:07:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Mar 18 20:07:22.443150565 (195h56m25.458820166s) [upload=188210]: Starting upload
Mar 18 20:07:22.443263267 (195h56m25.458932859s) [upload=188210]: Done starting upload
Mar 18 20:07:22.443340001 (195h56m25.459009595s) [upload=188210]: Uploading segments...
Mar 18 20:07:22.443603601 (195h56m25.459273202s) [upload=188210]: Got next segment. Inline: false
Mar 18 20:07:22.443650680 (195h56m25.459320305s) [upload=188210]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:07:22.443656248 (195h56m25.459325848s) [upload=188210]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:07:26.223437220 (195h56m29.239106823s) [seg_pos={0 0}, upload=188210]: Begin upload segment...
Mar 18 20:07:26.223501236 (195h56m29.239170827s) [seg_pos={0 0}, upload=188210]: Done begin upload segment.
Mar 18 20:07:26.223530102 (195h56m29.239199702s) [noise=true , address=145.79.227.44:4194 , piece_id=QB6PXLOBBNK4TDN55LTO3KCDSXV6WN5QO76F722H6Q5DJI7YXDTQ, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.223575490 (195h56m29.239245099s) [noise=true , address=145.79.227.10:4405 , piece_id=M6EIPV5YMLSPUHX6GWI66MQMPG5MCEPYFAP6QYE3HVVKRPFVSBZA, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.224134799 (195h56m29.239804407s) [noise=true , address=77.90.31.17:4770 , piece_id=7MATO45O7ESRBBVSRKZXJGAMEZHCNTCZ75ZEDZAOYHY3GCE65UQA, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.224669775 (195h56m29.240339376s) [noise=true , address=145.79.227.21:4476 , piece_id=Y33LPFUQPHHQY5OKKFJKHPCFAZVQEOJWCLWJZ5XGUUORLFOHH2IQ, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.225160763 (195h56m29.240830366s) [noise=true , address=77.90.31.13:4629 , piece_id=FQLZAXBBDLJ36JWYNEVMYGV37NB6Z2SZXDM4NXI5YQKWBVIRVRNQ, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.225469308 (195h56m29.241138917s) [noise=true , address=77.90.31.14:4665 , piece_id=MDGSIZWF6B6V7KXFYIXNOMBILPRGCMSQ2XCTB6N7QYWVSKBNOHQQ, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.226048238 (195h56m29.241717848s) [noise=true , address=45.91.116.7:4024 , piece_id=24TJPXYNVCC6CF3OUPDVPZ2QDL67O66O54RLXBXX2AVHP5GMXLZA, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.226150741 (195h56m29.241820344s) [noise=true , address=45.91.116.12:4225 , piece_id=E5YCTKHIFZGHEV23J27PSAJAIE7BW7H5I2KH6VX753IVFSP6WJPA, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.226460174 (195h56m29.242129785s) [noise=true , address=45.91.116.9:4261 , piece_id=NEPFK3HVGSNUIM7AZ4DHVGXXXZZFALKCRUYXLERWOIYAKM5DJ6RQ, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.226552421 (195h56m29.242222024s) [noise=true , address=45.91.116.14:4216 , piece_id=OTKU5MGVM7RMQNFKBBUT7DLCDRDBKSUMMQJXQW5EW3EKWHESIZOA, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.226950090 (195h56m29.242619697s) [noise=true , address=45.91.116.4:4952 , piece_id=UOSV6T52MK3LZMJMD6OJSWEH6DREL5OKGNZYRQ2VS5RQVYXZIBXA, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.227160634 (195h56m29.242830237s) [noise=true , address=45.91.116.6:4974 , piece_id=C4HX3Q75OTTADRVQDQZLGSPLP72ZT5RC3QVM2NWY2FY5B6LCNAVQ, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.227333105 (195h56m29.243002714s) [noise=true , address=45.91.116.10:4083 , piece_id=R7RDSPBGKFM3TXL5SEZVLSIDBN7KJ4UG3BV75L5UYKHXE7CZXAMQ, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.227608540 (195h56m29.243278144s) [noise=true , address=145.79.227.42:4121 , piece_id=4PJGVHDCWF4MRDLFB7VAC7DDDIRBDW4YDAV7SU7VVYLCQPRIQL4A, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.227723644 (195h56m29.24339325s) [noise=true , address=145.79.227.4:4188 , piece_id=NKRKEBJI2S5U3XSEYN4VHU7DFACDBNSY3TBXUXE3N2LSUPFM747A, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.228089272 (195h56m29.243758879s) [noise=true , address=45.91.116.11:4119 , piece_id=4GLR3XTOOZDDKU4DMWAYUNBSBPKQSGRVK2NTS37P5LCTJLRS7LOA, seg_pos={0 0}, upload=188210]: Uploading piece...
Mar 18 20:07:26.821097987 (195h56m29.836767587s) [upload=188210]: Next returned nil segment
Mar 18 20:07:26.821112732 (195h56m29.836782336s) [upload=188210]: Waiting for error group managing segments...
Mar 18 20:07:26.954231193 (195h56m29.969900793s) [noise=true , address=77.90.31.14:4665 , piece_id=MDGSIZWF6B6V7KXFYIXNOMBILPRGCMSQ2XCTB6N7QYWVSKBNOHQQ, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:26.966087375 (195h56m29.981756976s) [noise=true , address=77.90.31.13:4629 , piece_id=FQLZAXBBDLJ36JWYNEVMYGV37NB6Z2SZXDM4NXI5YQKWBVIRVRNQ, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:26.966168399 (195h56m29.981838012s) [noise=true , address=45.91.116.10:4083 , piece_id=R7RDSPBGKFM3TXL5SEZVLSIDBN7KJ4UG3BV75L5UYKHXE7CZXAMQ, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:26.967571593 (195h56m29.983241202s) [noise=true , address=45.91.116.12:4225 , piece_id=E5YCTKHIFZGHEV23J27PSAJAIE7BW7H5I2KH6VX753IVFSP6WJPA, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:26.979247785 (195h56m29.994917391s) [noise=true , address=77.90.31.17:4770 , piece_id=7MATO45O7ESRBBVSRKZXJGAMEZHCNTCZ75ZEDZAOYHY3GCE65UQA, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:26.979614836 (195h56m29.995284437s) [noise=true , address=45.91.116.14:4216 , piece_id=OTKU5MGVM7RMQNFKBBUT7DLCDRDBKSUMMQJXQW5EW3EKWHESIZOA, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:26.979969259 (195h56m29.995638862s) [noise=true , address=45.91.116.7:4024 , piece_id=24TJPXYNVCC6CF3OUPDVPZ2QDL67O66O54RLXBXX2AVHP5GMXLZA, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:26.982060753 (195h56m29.997730354s) [noise=true , address=45.91.116.4:4952 , piece_id=UOSV6T52MK3LZMJMD6OJSWEH6DREL5OKGNZYRQ2VS5RQVYXZIBXA, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:26.985561178 (195h56m30.001230796s) [noise=true , address=45.91.116.9:4261 , piece_id=NEPFK3HVGSNUIM7AZ4DHVGXXXZZFALKCRUYXLERWOIYAKM5DJ6RQ, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:26.987414041 (195h56m30.003083644s) [noise=true , address=45.91.116.11:4119 , piece_id=4GLR3XTOOZDDKU4DMWAYUNBSBPKQSGRVK2NTS37P5LCTJLRS7LOA, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:27.011123147 (195h56m30.026792751s) [noise=true , address=45.91.116.6:4974 , piece_id=C4HX3Q75OTTADRVQDQZLGSPLP72ZT5RC3QVM2NWY2FY5B6LCNAVQ, seg_pos={0 0}, upload=188210]: Done uploading piece. err: <nil>
Mar 18 20:07:52.146067078 (195h56m55.161736688s) [noise=true , address=145.79.227.10:4405 , piece_id=M6EIPV5YMLSPUHX6GWI66MQMPG5MCEPYFAP6QYE3HVVKRPFVSBZA, seg_pos={0 0}, upload=188210]: Done uploading piece. err: ecclient: upload failed (node:1MCHYW29ZQjE2fHN8P98hDfNRLsjiVqabF86TXz5nZRBPhbGy4, address:145.79.227.10:4405): protocol: rpc: dial tcp 145.79.227.10:4405: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:07:52.146374155 (195h56m55.162043761s) [noise=true , address=145.79.227.21:4476 , piece_id=Y33LPFUQPHHQY5OKKFJKHPCFAZVQEOJWCLWJZ5XGUUORLFOHH2IQ, seg_pos={0 0}, upload=188210]: Done uploading piece. err: ecclient: upload failed (node:12NKwRvjiEpY7DrbqFGaDHQ8no5YV4cUALnpsvQ8icUPoLvMYen, address:145.79.227.21:4476): protocol: rpc: dial tcp 145.79.227.21:4476: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:07:52.146628665 (195h56m55.162298269s) [noise=true , address=145.79.227.4:4188 , piece_id=NKRKEBJI2S5U3XSEYN4VHU7DFACDBNSY3TBXUXE3N2LSUPFM747A, seg_pos={0 0}, upload=188210]: Done uploading piece. err: ecclient: upload failed (node:12pLFHQ2Xj6n53w9eLQtMB4A29k3Xh6w716z5s2CGxwPnBoZcgZ, address:145.79.227.4:4188): protocol: rpc: dial tcp 145.79.227.4:4188: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:07:52.146804270 (195h56m55.162473879s) [noise=true , address=145.79.227.44:4194 , piece_id=QB6PXLOBBNK4TDN55LTO3KCDSXV6WN5QO76F722H6Q5DJI7YXDTQ, seg_pos={0 0}, upload=188210]: Done uploading piece. err: ecclient: upload failed (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh, address:145.79.227.44:4194): protocol: rpc: dial tcp 145.79.227.44:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:07:52.146954562 (195h56m55.162624168s) [noise=true , address=145.79.227.42:4121 , piece_id=4PJGVHDCWF4MRDLFB7VAC7DDDIRBDW4YDAV7SU7VVYLCQPRIQL4A, seg_pos={0 0}, upload=188210]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:07:52.146979033 (195h56m55.162648619s) [seg_pos={0 0}, upload=188210]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Mar 18 20:07:52.257084603 (195h56m55.272754197s) [upload=188210]: Done uploading segments...
Mar 18 20:07:52.257134168 (195h56m55.272803769s) [upload=188210]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.113398419s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [..............XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001432024s trace:
https://jaeger.dev.storj.io/trace/410e9185d9e8a8b1
FAIL (2026-03-18 20:07:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001452489s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [....................XXX.XXX...............X.................] duration:
6.346967939s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:07:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.346998605s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [....................X..X...X..X..X........X..X..X...X.......] duration:
18.041215458s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:07:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 18.041241907s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.....................XXX.XXX...............X................] duration:
19.613321003s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:07:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.613405525s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [...............XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000843087s trace:
https://jaeger.dev.storj.io/trace/54a18cd3b96865a8
FAIL (2026-03-18 20:07:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000870622s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
41 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration:
30.121143385s trace:
https://jaeger.dev.storj.io/trace/36d60e21d971db05
FAIL (2026-03-18 20:06:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
gments...
Mar 18 20:06:29.475763940 (195h55m32.491433544s) [upload=188194]: Got next segment. Inline: false
Mar 18 20:06:29.475839691 (195h55m32.491509291s) [upload=188194]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:06:29.475847143 (195h55m32.491516744s) [upload=188194]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:06:33.031667477 (195h55m36.047337088s) [seg_pos={0 0}, upload=188194]: Begin upload segment...
Mar 18 20:06:33.031750321 (195h55m36.047419925s) [seg_pos={0 0}, upload=188194]: Done begin upload segment.
Mar 18 20:06:33.031805266 (195h55m36.047474878s) [noise=true , address=77.90.31.16:4738 , piece_id=OAMSAC4IWSPCZ6WDZOVYD5Q4LW2X7GKY7PRMDGUE4OS74NGS7LBQ, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.036871838 (195h55m36.05254145s) [noise=true , address=45.91.116.11:4118 , piece_id=RG67BH23XEFCEQVEXWU2437XBV63WR7KXSU5IV6M4U6JHT7ATPCQ, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.041243741 (195h55m36.056913359s) [noise=true , address=45.91.116.5:4300 , piece_id=RGSOEXNCPAKHPQS4RUW27CPPVMQU6SFBG3CJEXT3FYFAVJV2CRWA, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.043520569 (195h55m36.059190181s) [noise=true , address=145.79.227.24:4585 , piece_id=Q7IYHZ2UY7CFSG5T3F5JRMKTIMSF5DQ7246XCIG2XLL3ZYR62QLA, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.045344146 (195h55m36.061013758s) [noise=true , address=79.127.219.38:4822 , piece_id=XXLFUS7RYFFQV3U2BMSBJKY3JPNUPJDHUFTA4RCOY25J2EDHJ3XA, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.049597629 (195h55m36.065267251s) [noise=true , address=145.79.227.8:4332 , piece_id=MDVOCH5JPF5FQW4WQ5THDNYDV4QVESMMQJK7YD6LMNDQOUOT5I6Q, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.051451286 (195h55m36.067120896s) [noise=true , address=145.79.227.21:4477 , piece_id=R3QYITAS7BQC62IJPI45CDLTXJ4XVZ6PAMDNYVNRIQ32P6LGBOSA, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.054675540 (195h55m36.070345156s) [noise=true , address=145.79.227.44:4193 , piece_id=H7KOYN6UDWVQMUHLIYXFTEWQJKYRUW7PUYBI6VCYGQOANYBUNCRQ, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.059777889 (195h55m36.075447505s) [noise=true , address=145.79.227.33:4869 , piece_id=FDYFRPUM4GE3LY23XJNYTFLRDMAUGYSF57I5H7YARPBMIPDSXWYA, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.060089743 (195h55m36.075759361s) [noise=true , address=77.90.31.17:4774 , piece_id=34MGXGN2P2UZ27SA3W46E3ZV5Y4BZ2VEOIPS2EQPLTWC5WLLBXEA, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.060263437 (195h55m36.075933045s) [noise=true , address=145.79.227.23:4548 , piece_id=QF6MPLBVWHEXWXNHHX7AWTIZIS4XSWOZRL4RNC6XJQHZINDFVTMA, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.065694420 (195h55m36.081364034s) [noise=true , address=145.79.227.7:4296 , piece_id=2Q5MTWULHO42ROAGQ2Q54QRPPJ7ZVRREZC2Q347MSTM65U6Q533Q, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.067896375 (195h55m36.083565993s) [noise=true , address=79.127.219.45:4858 , piece_id=QTZI3LH2D444UC7MQJ4RMDKHJAX5P6JNDECH2V7KRTGV23GAQM2Q, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.068222817 (195h55m36.08389243s) [noise=true , address=45.91.116.13:4157 , piece_id=E7JF37H3RF4ICUNEOCYY7QQXVR3PP6A46WKOW6AKVTGEERCA2MZA, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.075164603 (195h55m36.090834227s) [noise=true , address=145.79.227.26:4656 , piece_id=HIOBZX3KMSYVLEOWXSS7PAEGLSEKGPCMLXEKZFIQXCRYD7NMSZIA, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.079350636 (195h55m36.095020262s) [noise=true , address=145.79.227.9:4369 , piece_id=JKV4B7ORQGJF2IZRCWNQZWOTR6ZQZ67REE4VFBBQ5OJKM6WDSZCQ, seg_pos={0 0}, upload=188194]: Uploading piece...
Mar 18 20:06:33.662673872 (195h55m36.678343485s) [upload=188194]: Next returned nil segment
Mar 18 20:06:33.662698542 (195h55m36.678368154s) [upload=188194]: Waiting for error group managing segments...
Mar 18 20:06:33.796136544 (195h55m36.811806145s) [noise=true , address=77.90.31.16:4738 , piece_id=OAMSAC4IWSPCZ6WDZOVYD5Q4LW2X7GKY7PRMDGUE4OS74NGS7LBQ, seg_pos={0 0}, upload=188194]: Done uploading piece. err: <nil>
Mar 18 20:06:33.814305344 (195h55m36.829974971s) [noise=true , address=45.91.116.13:4157 , piece_id=E7JF37H3RF4ICUNEOCYY7QQXVR3PP6A46WKOW6AKVTGEERCA2MZA, seg_pos={0 0}, upload=188194]: Done uploading piece. err: <nil>
Mar 18 20:06:33.818047427 (195h55m36.833717033s) [noise=true , address=45.91.116.11:4118 , piece_id=RG67BH23XEFCEQVEXWU2437XBV63WR7KXSU5IV6M4U6JHT7ATPCQ, seg_pos={0 0}, upload=188194]: Done uploading piece. err: <nil>
Mar 18 20:06:33.822233436 (195h55m36.837903042s) [noise=true , address=77.90.31.17:4774 , piece_id=34MGXGN2P2UZ27SA3W46E3ZV5Y4BZ2VEOIPS2EQPLTWC5WLLBXEA, seg_pos={0 0}, upload=188194]: Done uploading piece. err: <nil>
Mar 18 20:06:33.823959089 (195h55m36.839628689s) [noise=true , address=45.91.116.5:4300 , piece_id=RGSOEXNCPAKHPQS4RUW27CPPVMQU6SFBG3CJEXT3FYFAVJV2CRWA, seg_pos={0 0}, upload=188194]: Done uploading piece. err: <nil>
Mar 18 20:06:33.843684300 (195h55m36.859353908s) [noise=true , address=79.127.219.45:4858 , piece_id=QTZI3LH2D444UC7MQJ4RMDKHJAX5P6JNDECH2V7KRTGV23GAQM2Q, seg_pos={0 0}, upload=188194]: Done uploading piece. err: <nil>
Mar 18 20:06:33.867190122 (195h55m36.882859729s) [noise=true , address=79.127.219.38:4822 , piece_id=XXLFUS7RYFFQV3U2BMSBJKY3JPNUPJDHUFTA4RCOY25J2EDHJ3XA, seg_pos={0 0}, upload=188194]: Done uploading piece. err: <nil>
Mar 18 20:06:59.172480698 (195h56m2.188150321s) [noise=true , address=145.79.227.33:4869 , piece_id=FDYFRPUM4GE3LY23XJNYTFLRDMAUGYSF57I5H7YARPBMIPDSXWYA, seg_pos={0 0}, upload=188194]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:145.79.227.33:4869): protocol: rpc: dial tcp 145.79.227.33:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:59.172793607 (195h56m2.188463217s) [noise=true , address=145.79.227.26:4656 , piece_id=HIOBZX3KMSYVLEOWXSS7PAEGLSEKGPCMLXEKZFIQXCRYD7NMSZIA, seg_pos={0 0}, upload=188194]: Done uploading piece. err: ecclient: upload failed (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j, address:145.79.227.26:4656): protocol: rpc: dial tcp 145.79.227.26:4656: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:59.173059170 (195h56m2.188728777s) [noise=true , address=145.79.227.21:4477 , piece_id=R3QYITAS7BQC62IJPI45CDLTXJ4XVZ6PAMDNYVNRIQ32P6LGBOSA, seg_pos={0 0}, upload=188194]: Done uploading piece. err: ecclient: upload failed (node:1QNztqPaSGyZVsQihaj9YjG9C1nzKi1crfSDjDHimYboYAKWZY, address:145.79.227.21:4477): protocol: rpc: dial tcp 145.79.227.21:4477: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:59.173260808 (195h56m2.188930417s) [noise=true , address=145.79.227.24:4585 , piece_id=Q7IYHZ2UY7CFSG5T3F5JRMKTIMSF5DQ7246XCIG2XLL3ZYR62QLA, seg_pos={0 0}, upload=188194]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:145.79.227.24:4585): protocol: rpc: dial tcp 145.79.227.24:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:59.174995895 (195h56m2.190665504s) [noise=true , address=145.79.227.7:4296 , piece_id=2Q5MTWULHO42ROAGQ2Q54QRPPJ7ZVRREZC2Q347MSTM65U6Q533Q, seg_pos={0 0}, upload=188194]: Done uploading piece. err: ecclient: upload failed (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1, address:145.79.227.7:4296): protocol: rpc: dial tcp 145.79.227.7:4296: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:59.175450815 (195h56m2.191120424s) [noise=true , address=145.79.227.23:4548 , piece_id=QF6MPLBVWHEXWXNHHX7AWTIZIS4XSWOZRL4RNC6XJQHZINDFVTMA, seg_pos={0 0}, upload=188194]: Done uploading piece. err: ecclient: upload failed (node:19rx4nU8dUTtPXvMonzZSBkn5Z5cnBXrndvU7t8ewUmfYLBqid, address:145.79.227.23:4548): protocol: rpc: dial tcp 145.79.227.23:4548: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:59.175723521 (195h56m2.191393145s) [noise=true , address=145.79.227.8:4332 , piece_id=MDVOCH5JPF5FQW4WQ5THDNYDV4QVESMMQJK7YD6LMNDQOUOT5I6Q, seg_pos={0 0}, upload=188194]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: rpc: dial tcp 145.79.227.8:4332: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:59.175905073 (195h56m2.191574675s) [noise=true , address=145.79.227.44:4193 , piece_id=H7KOYN6UDWVQMUHLIYXFTEWQJKYRUW7PUYBI6VCYGQOANYBUNCRQ, seg_pos={0 0}, upload=188194]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:145.79.227.44:4193): protocol: rpc: dial tcp 145.79.227.44:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:59.175948105 (195h56m2.191617698s) [noise=true , address=145.79.227.9:4369 , piece_id=JKV4B7ORQGJF2IZRCWNQZWOTR6ZQZ67REE4VFBBQ5OJKM6WDSZCQ, seg_pos={0 0}, upload=188194]: Done uploading piece. err: ecclient: upload failed (node:1nwygPHyRJEZkjCoSZJTVhXDKLHc7AonVMzKMht6ppwZwhcGCT, address:145.79.227.9:4369): protocol: rpc: dial tcp 145.79.227.9:4369: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:59.175990747 (195h56m2.191660347s) [seg_pos={0 0}, upload=188194]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Mar 18 20:06:59.291587159 (195h56m2.307256761s) [upload=188194]: Done uploading segments...
Mar 18 20:06:59.291640009 (195h56m2.307309612s) [upload=188194]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 30.121171703s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.....................X..X...X..X..X........X..X..X...X......] duration:
21.548667249s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:06:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 21.548700562s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001355944s trace:
https://jaeger.dev.storj.io/trace/583e6120b0c51802
FAIL (2026-03-18 20:06:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001379542s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
40 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration:
30.12165929s trace:
https://jaeger.dev.storj.io/trace/75458a3ecf3063b4
FAIL (2026-03-18 20:06:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Mar 18 20:05:49.552079194 (195h54m52.567748795s) [upload=188186]: Starting upload
Mar 18 20:05:49.552174833 (195h54m52.567844443s) [upload=188186]: Done starting upload
Mar 18 20:05:49.552327834 (195h54m52.567997422s) [upload=188186]: Uploading segments...
Mar 18 20:05:49.552498249 (195h54m52.568167853s) [upload=188186]: Got next segment. Inline: false
Mar 18 20:05:49.552547523 (195h54m52.568217123s) [upload=188186]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:05:49.552555783 (195h54m52.568225394s) [upload=188186]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:05:53.324656107 (195h54m56.340325698s) [seg_pos={0 0}, upload=188186]: Begin upload segment...
Mar 18 20:05:53.324756551 (195h54m56.340426155s) [noise=true , address=77.90.31.12:4589 , piece_id=KJNF6MQH7ORC7FJLGJ566JE42XXXW3FCL5LX7J6A5T6VQAKXL6DQ, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.324802806 (195h54m56.340472406s) [seg_pos={0 0}, upload=188186]: Done begin upload segment.
Mar 18 20:05:53.324824056 (195h54m56.340493658s) [noise=true , address=77.90.31.17:4770 , piece_id=LPUM7LIHZYL6SS2MSU5IHVFK7ZVQ35VUFO7OXDULQRI2JVLZIMQA, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.325801765 (195h54m56.341471373s) [noise=true , address=79.127.219.45:4858 , piece_id=AAE4OPJKOJQD3PZTL2HR4L3ILKRM4CDCBM7J3AQTTSJ5BNICCRVQ, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.326449057 (195h54m56.342118646s) [noise=true , address=79.127.219.41:4930 , piece_id=KLEPQYBP3ZEJL7H6KBQ2GZ4BTG7Z5LJOVEMGIIADIK4AVKBUPMTQ, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.327227511 (195h54m56.34289712s) [noise=true , address=45.91.116.11:4117 , piece_id=RQU24KDONDWEWZSSWF6EU2WWW7ZJS53KJ4NPQVT5FEGJNTRECWDA, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.327868477 (195h54m56.343538082s) [noise=true , address=77.90.31.16:4738 , piece_id=HNHK4QX4XDWLOKACPMY64KMZEAGRRDDEFBQI63BLYT6CHD45YNRA, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.328866458 (195h54m56.344536067s) [noise=true , address=145.79.227.26:4658 , piece_id=BKC52FLE32BP34TDEFFVGXJSH5KKW4BDXD5TYLIJOIXW6R3SICOQ, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.329401873 (195h54m56.345071478s) [noise=true , address=145.79.227.34:4908 , piece_id=JWUJUWP3BOZINOQY5C5XCKEWQWJB52U4ERYHK5TRSZMZZ55KXBFQ, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.330054658 (195h54m56.34572426s) [noise=true , address=77.90.31.14:4665 , piece_id=4FLZAWCSMWIIRP7WUSL7UHNGZSX5W777ONPH3PETDJA3D7EPS2FA, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.331257780 (195h54m56.346927383s) [noise=true , address=145.79.227.3:4155 , piece_id=7FPBBHZSCQCRWCMGLQSYZ2HO223LPZ2WW4IENEM2GDVRIVQS7XIA, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.332715330 (195h54m56.348384932s) [noise=true , address=45.91.116.12:4228 , piece_id=FOHHEPJ4SJGPZ2CSBQSI5C75O25E6XEWQUU6HLDISSIIALZUU3UQ, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.335116365 (195h54m56.350785972s) [noise=true , address=145.79.227.43:4159 , piece_id=RRH45HVWSKMOXX3BGKXHKMWMLASUDPTB54C4LWANEBND6OMJAQ4Q, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.336578420 (195h54m56.352248022s) [noise=true , address=45.91.116.7:4026 , piece_id=5RJTKA3TLZYBVOTZUXOZIJEKSK4QOEZZ4WPJ3KI7DIHWYWRNQSDA, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.338305413 (195h54m56.353975025s) [noise=true , address=45.91.116.14:4224 , piece_id=OW6JNZ3CVJF24H6IPZPSWVZ7FEPRB37J33YYYFK3E6X4FD5BDRTA, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.339802273 (195h54m56.355471865s) [noise=true , address=45.91.116.8:4070 , piece_id=NDXPB2OG3YO24CTZ4N32RZVUXIDWHAPRCLPR7UCVZAHZS3SLFNXQ, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.341248105 (195h54m56.356917709s) [noise=true , address=145.79.227.37:4567 , piece_id=KQG6HW2TCCI3X33JNBV2FTZ4JTJKNVIM5T5R5YW6J4HUODQBPPFA, seg_pos={0 0}, upload=188186]: Uploading piece...
Mar 18 20:05:53.939068705 (195h54m56.954738293s) [upload=188186]: Next returned nil segment
Mar 18 20:05:53.939081900 (195h54m56.954751499s) [upload=188186]: Waiting for error group managing segments...
Mar 18 20:05:54.069972691 (195h54m57.085642294s) [noise=true , address=77.90.31.16:4738 , piece_id=HNHK4QX4XDWLOKACPMY64KMZEAGRRDDEFBQI63BLYT6CHD45YNRA, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.070525668 (195h54m57.086195269s) [noise=true , address=45.91.116.11:4117 , piece_id=RQU24KDONDWEWZSSWF6EU2WWW7ZJS53KJ4NPQVT5FEGJNTRECWDA, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.072379192 (195h54m57.088048797s) [noise=true , address=77.90.31.12:4589 , piece_id=KJNF6MQH7ORC7FJLGJ566JE42XXXW3FCL5LX7J6A5T6VQAKXL6DQ, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.073106288 (195h54m57.088775889s) [noise=true , address=77.90.31.14:4665 , piece_id=4FLZAWCSMWIIRP7WUSL7UHNGZSX5W777ONPH3PETDJA3D7EPS2FA, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.076632652 (195h54m57.092302253s) [noise=true , address=45.91.116.14:4224 , piece_id=OW6JNZ3CVJF24H6IPZPSWVZ7FEPRB37J33YYYFK3E6X4FD5BDRTA, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.078657498 (195h54m57.094327099s) [noise=true , address=45.91.116.12:4228 , piece_id=FOHHEPJ4SJGPZ2CSBQSI5C75O25E6XEWQUU6HLDISSIIALZUU3UQ, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.081998210 (195h54m57.097667813s) [noise=true , address=79.127.219.45:4858 , piece_id=AAE4OPJKOJQD3PZTL2HR4L3ILKRM4CDCBM7J3AQTTSJ5BNICCRVQ, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.082439310 (195h54m57.098108911s) [noise=true , address=45.91.116.7:4026 , piece_id=5RJTKA3TLZYBVOTZUXOZIJEKSK4QOEZZ4WPJ3KI7DIHWYWRNQSDA, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.088382299 (195h54m57.104051901s) [noise=true , address=77.90.31.17:4770 , piece_id=LPUM7LIHZYL6SS2MSU5IHVFK7ZVQ35VUFO7OXDULQRI2JVLZIMQA, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.098217258 (195h54m57.113886876s) [noise=true , address=45.91.116.8:4070 , piece_id=NDXPB2OG3YO24CTZ4N32RZVUXIDWHAPRCLPR7UCVZAHZS3SLFNXQ, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:05:54.180830796 (195h54m57.196500414s) [noise=true , address=79.127.219.41:4930 , piece_id=KLEPQYBP3ZEJL7H6KBQ2GZ4BTG7Z5LJOVEMGIIADIK4AVKBUPMTQ, seg_pos={0 0}, upload=188186]: Done uploading piece. err: <nil>
Mar 18 20:06:19.246500969 (195h55m22.262170582s) [noise=true , address=145.79.227.26:4658 , piece_id=BKC52FLE32BP34TDEFFVGXJSH5KKW4BDXD5TYLIJOIXW6R3SICOQ, seg_pos={0 0}, upload=188186]: Done uploading piece. err: ecclient: upload failed (node:1gcSHRechDzTDwivRoAwNFfNj5Zjqvi6oJwVqzbCQinwgWtpPr, address:145.79.227.26:4658): protocol: rpc: dial tcp 145.79.227.26:4658: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:19.249245414 (195h55m22.264915025s) [noise=true , address=145.79.227.37:4567 , piece_id=KQG6HW2TCCI3X33JNBV2FTZ4JTJKNVIM5T5R5YW6J4HUODQBPPFA, seg_pos={0 0}, upload=188186]: Done uploading piece. err: ecclient: upload failed (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS, address:145.79.227.37:4567): protocol: rpc: dial tcp 145.79.227.37:4567: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:19.249899117 (195h55m22.265568725s) [noise=true , address=145.79.227.3:4155 , piece_id=7FPBBHZSCQCRWCMGLQSYZ2HO223LPZ2WW4IENEM2GDVRIVQS7XIA, seg_pos={0 0}, upload=188186]: Done uploading piece. err: ecclient: upload failed (node:1yiUBiCEpAs5yDmgVfFGuxgW4R53jzMJugo8Piff3p79EEZdWr, address:145.79.227.3:4155): protocol: rpc: dial tcp 145.79.227.3:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:19.250398899 (195h55m22.266068513s) [noise=true , address=145.79.227.43:4159 , piece_id=RRH45HVWSKMOXX3BGKXHKMWMLASUDPTB54C4LWANEBND6OMJAQ4Q, seg_pos={0 0}, upload=188186]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: rpc: dial tcp 145.79.227.43:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:19.250785862 (195h55m22.266455473s) [noise=true , address=145.79.227.34:4908 , piece_id=JWUJUWP3BOZINOQY5C5XCKEWQWJB52U4ERYHK5TRSZMZZ55KXBFQ, seg_pos={0 0}, upload=188186]: Done uploading piece. err: ecclient: upload failed (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2, address:145.79.227.34:4908): protocol: rpc: dial tcp 145.79.227.34:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:06:19.250825920 (195h55m22.266495524s) [seg_pos={0 0}, upload=188186]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Mar 18 20:06:19.364868384 (195h55m22.380537994s) [upload=188186]: Done uploading segments...
Mar 18 20:06:19.364933643 (195h55m22.38060325s) [upload=188186]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.121676598s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [.................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000973952s trace:
https://jaeger.dev.storj.io/trace/4105888c1398d8e4
FAIL (2026-03-18 20:06:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000997446s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [......................XXX.XXX...............X...............] duration:
18.999407434s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:06:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.999440283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
39 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration:
30.118331187s trace:
https://jaeger.dev.storj.io/trace/633469486f240b2d
FAIL (2026-03-18 20:05:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Mar 18 20:05:19.294230508 (195h54m22.30990012s) [upload=188178]: Starting upload
Mar 18 20:05:19.294335501 (195h54m22.310005114s) [upload=188178]: Done starting upload
Mar 18 20:05:19.294600878 (195h54m22.310270484s) [upload=188178]: Uploading segments...
Mar 18 20:05:19.297233369 (195h54m22.312902984s) [upload=188178]: Got next segment. Inline: false
Mar 18 20:05:19.298965108 (195h54m22.314634721s) [upload=188178]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:05:19.298984969 (195h54m22.314654583s) [upload=188178]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:05:22.583364741 (195h54m25.599034351s) [seg_pos={0 0}, upload=188178]: Begin upload segment...
Mar 18 20:05:22.583439426 (195h54m25.599109036s) [seg_pos={0 0}, upload=188178]: Done begin upload segment.
Mar 18 20:05:22.583490130 (195h54m25.59915974s) [noise=true , address=145.79.227.40:4052 , piece_id=VIXE6YLII6SOLI7HMHGJLB5IIHXIRV7BOUIM3GO5FIOYBQ62Z2CQ, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.583575475 (195h54m25.599245087s) [noise=true , address=79.127.219.38:4820 , piece_id=V3IZKUQDUEK6NOPBDDDF3OOYV743K3AX5JUGJJIRPVDD4IMSXD6Q, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.584294977 (195h54m25.599964587s) [noise=true , address=145.79.227.36:4521 , piece_id=WCY646LF6AP5LEUTBLGOEBFEVQXLTYBJRI67YJGIWELWT5Q7WK2Q, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.584376895 (195h54m25.6000465s) [noise=true , address=77.90.31.13:4630 , piece_id=UESX5XPGPVELYV7M2RJA5WADN6V65XVUKYVRDLRJSWOZH6I4GAPA, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.585392443 (195h54m25.601062051s) [noise=true , address=145.79.227.10:4407 , piece_id=4DVLSSYKOJEGPQKTCIII4PYD4JRLTPMLKLZ2NJ5LFGTIAKNXTYTQ, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.586113622 (195h54m25.601783227s) [noise=true , address=45.91.116.6:4989 , piece_id=XRUUCS7RQRDD4IFK6HO4AZ3ONV2GXA7HRTZVJEVSRCEPJKQZ6F7Q, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.586167546 (195h54m25.601837146s) [noise=true , address=77.90.31.19:4844 , piece_id=IQZ54RHKHS7CYIBREYZEYFYR2HKD534Z3ICO3MUXDQK2D6MKRKXQ, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.586632062 (195h54m25.602301666s) [noise=true , address=145.79.227.28:4730 , piece_id=2ZC2ZQJZCJ3EHJNHZIL7QQ6XL6T4BD3Y7CHI26SL2ON42SVXI56A, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.586767618 (195h54m25.602437227s) [noise=true , address=77.90.31.14:4665 , piece_id=RDNXMA64JA4I2HPURMLQWHOQL4GLORYUV5JD2LZZPMQQP44R26AA, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.587151123 (195h54m25.602820723s) [noise=true , address=45.91.116.4:4940 , piece_id=CC7FCPY343M2HV3F72IKFI4MI5AHF6W26DONDYQSYQMGASFZVTSQ, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.587435095 (195h54m25.603104704s) [noise=true , address=77.90.31.11:4555 , piece_id=LSKISAPHX6WXWHAHGTLGCB4TXHZYCL2TQGAL4V2326PWP73ASZ4Q, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.587620913 (195h54m25.603290529s) [noise=true , address=145.79.227.41:4087 , piece_id=ONVFR4RJG6S3ZXZ5RYJYCAAHY2GFYPQXM3RGYU4T4IY2PVO2ORVQ, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.588116725 (195h54m25.603786326s) [noise=true , address=45.91.116.12:4228 , piece_id=RMUTVVTHDZTOGIUZYQS7KFTNBZDWDKTMCOYD5E3SI5O33O7J5VCQ, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.588139145 (195h54m25.603808757s) [noise=true , address=79.127.219.41:4929 , piece_id=OCV7ZZQ4WTMYXIWUE5354HH54DS5TDHPV3SOOWGWGT5Y4CZXDOUA, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.590007075 (195h54m25.605676678s) [noise=true , address=145.79.227.25:4619 , piece_id=EIM3UOL6CIRLKWRKDXTXH7OBUB7LXQATOJ2QMAENM6WZMBH3JT4A, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:22.590842726 (195h54m25.606512333s) [noise=true , address=45.91.116.14:4192 , piece_id=ED7GAHJN35Q56GIOJAARTNERV6M23DW5XDVGGQBGNUBZ7I425UHQ, seg_pos={0 0}, upload=188178]: Uploading piece...
Mar 18 20:05:23.202696691 (195h54m26.218366293s) [upload=188178]: Next returned nil segment
Mar 18 20:05:23.202715415 (195h54m26.218385028s) [upload=188178]: Waiting for error group managing segments...
Mar 18 20:05:23.333016691 (195h54m26.348686292s) [noise=true , address=77.90.31.11:4555 , piece_id=LSKISAPHX6WXWHAHGTLGCB4TXHZYCL2TQGAL4V2326PWP73ASZ4Q, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:23.346776022 (195h54m26.362445629s) [noise=true , address=45.91.116.12:4228 , piece_id=RMUTVVTHDZTOGIUZYQS7KFTNBZDWDKTMCOYD5E3SI5O33O7J5VCQ, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:23.352060749 (195h54m26.36773035s) [noise=true , address=45.91.116.14:4192 , piece_id=ED7GAHJN35Q56GIOJAARTNERV6M23DW5XDVGGQBGNUBZ7I425UHQ, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:23.363030383 (195h54m26.378699984s) [noise=true , address=45.91.116.6:4989 , piece_id=XRUUCS7RQRDD4IFK6HO4AZ3ONV2GXA7HRTZVJEVSRCEPJKQZ6F7Q, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:23.363880168 (195h54m26.379549771s) [noise=true , address=45.91.116.4:4940 , piece_id=CC7FCPY343M2HV3F72IKFI4MI5AHF6W26DONDYQSYQMGASFZVTSQ, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:23.387206608 (195h54m26.4028762s) [noise=true , address=79.127.219.38:4820 , piece_id=V3IZKUQDUEK6NOPBDDDF3OOYV743K3AX5JUGJJIRPVDD4IMSXD6Q, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:23.392034737 (195h54m26.407704337s) [noise=true , address=79.127.219.41:4929 , piece_id=OCV7ZZQ4WTMYXIWUE5354HH54DS5TDHPV3SOOWGWGT5Y4CZXDOUA, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:23.402191085 (195h54m26.417860686s) [noise=true , address=77.90.31.14:4665 , piece_id=RDNXMA64JA4I2HPURMLQWHOQL4GLORYUV5JD2LZZPMQQP44R26AA, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:23.409137262 (195h54m26.424806869s) [noise=true , address=77.90.31.19:4844 , piece_id=IQZ54RHKHS7CYIBREYZEYFYR2HKD534Z3ICO3MUXDQK2D6MKRKXQ, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:23.422663781 (195h54m26.438333381s) [noise=true , address=77.90.31.13:4630 , piece_id=UESX5XPGPVELYV7M2RJA5WADN6V65XVUKYVRDLRJSWOZH6I4GAPA, seg_pos={0 0}, upload=188178]: Done uploading piece. err: <nil>
Mar 18 20:05:48.968999993 (195h54m51.984669605s) [noise=true , address=145.79.227.40:4052 , piece_id=VIXE6YLII6SOLI7HMHGJLB5IIHXIRV7BOUIM3GO5FIOYBQ62Z2CQ, seg_pos={0 0}, upload=188178]: Done uploading piece. err: ecclient: upload failed (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V, address:145.79.227.40:4052): protocol: rpc: dial tcp 145.79.227.40:4052: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:48.969338259 (195h54m51.985007869s) [noise=true , address=145.79.227.10:4407 , piece_id=4DVLSSYKOJEGPQKTCIII4PYD4JRLTPMLKLZ2NJ5LFGTIAKNXTYTQ, seg_pos={0 0}, upload=188178]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:48.969384893 (195h54m51.985054501s) [noise=true , address=145.79.227.36:4521 , piece_id=WCY646LF6AP5LEUTBLGOEBFEVQXLTYBJRI67YJGIWELWT5Q7WK2Q, seg_pos={0 0}, upload=188178]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:145.79.227.36:4521): protocol: rpc: dial tcp 145.79.227.36:4521: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:48.969879488 (195h54m51.985549097s) [noise=true , address=145.79.227.28:4730 , piece_id=2ZC2ZQJZCJ3EHJNHZIL7QQ6XL6T4BD3Y7CHI26SL2ON42SVXI56A, seg_pos={0 0}, upload=188178]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: rpc: dial tcp 145.79.227.28:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:48.970093817 (195h54m51.985763427s) [noise=true , address=145.79.227.25:4619 , piece_id=EIM3UOL6CIRLKWRKDXTXH7OBUB7LXQATOJ2QMAENM6WZMBH3JT4A, seg_pos={0 0}, upload=188178]: Done uploading piece. err: ecclient: upload failed (node:1zuLJznAUFHkanuomswSd7JN6CkqPXso8Ak8e5E57eaEqvjodU, address:145.79.227.25:4619): protocol: rpc: dial tcp 145.79.227.25:4619: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:48.970312103 (195h54m51.985981718s) [noise=true , address=145.79.227.41:4087 , piece_id=ONVFR4RJG6S3ZXZ5RYJYCAAHY2GFYPQXM3RGYU4T4IY2PVO2ORVQ, seg_pos={0 0}, upload=188178]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: rpc: dial tcp 145.79.227.41:4087: i/o timeout; conn closed: op
Mar 18 20:05:48.970350348 (195h54m51.986019955s) [seg_pos={0 0}, upload=188178]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Mar 18 20:05:49.084368270 (195h54m52.10003787s) [upload=188178]: Done uploading segments...
Mar 18 20:05:49.084426627 (195h54m52.100096227s) [upload=188178]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.118348726s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.......................XXX.XXX...............X..............] duration:
18.593944482s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:05:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 18.59399171s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [......................X..X...X..X..X........X..X..X...X.....] duration:
14.311723961s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:05:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 14.311752199s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
38 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration:
30.115904087s trace:
https://jaeger.dev.storj.io/trace/342591bf1ecc29f3
FAIL (2026-03-18 20:05:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 7)
======LOGS======
ts...
Mar 18 20:04:35.430761951 (195h53m38.446431551s) [upload=188167]: Got next segment. Inline: false
Mar 18 20:04:35.430815563 (195h53m38.446485163s) [upload=188167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:04:35.430821706 (195h53m38.446491305s) [upload=188167]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:04:37.869828915 (195h53m40.885498517s) [seg_pos={0 0}, upload=188167]: Begin upload segment...
Mar 18 20:04:37.869972144 (195h53m40.885641746s) [seg_pos={0 0}, upload=188167]: Done begin upload segment.
Mar 18 20:04:37.870022356 (195h53m40.885691966s) [noise=true , address=145.79.227.49:4342 , piece_id=RHFVXERXH4A2F4ZY6BUO2VZ224F4MBNEU6OFJ7LFSTK7LCQ2DQ5Q, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.870820585 (195h53m40.886490191s) [noise=true , address=79.127.219.38:4821 , piece_id=4IMEOEQEKC2SFBU7IMU6Y4R4YZKL4USZNCNCGC3YQGNWVFFLJCHQ, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.871783149 (195h53m40.887452755s) [noise=true , address=45.91.116.13:4156 , piece_id=3RKQNBDJZ5VF6AFPMCPVKUUNPQXBIHQ6ULFKFF2JMOMGX6GI6LYA, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.872067599 (195h53m40.88773723s) [noise=true , address=145.79.227.20:4440 , piece_id=CKC5YGI7GRI6IMZBYGQS3ZJ6L5TLMCGE5JELGZVLC6UQAWMJTA5A, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.872234706 (195h53m40.887904307s) [noise=true , address=145.79.227.42:4122 , piece_id=Y5UPWO4HDUXPLLXJKIDZ7NGZXUPEMYNVLISS5ESRBKWZQ7E4UGBA, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.872463674 (195h53m40.888133277s) [noise=true , address=45.91.116.6:4975 , piece_id=47T5WWY2MM4QFLZQGDFJXGE6OJUSE5RQSRAVSFJTZ6HC2J3LC5RA, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.872656138 (195h53m40.88832575s) [noise=true , address=45.91.116.9:4296 , piece_id=KIPFBZRWSRPNLSUORGP23GILTRDKOL45DDZ45ZD4GROHGLB6LGYQ, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.873115277 (195h53m40.888784881s) [noise=true , address=145.79.227.33:4870 , piece_id=CGO24G7TIG6FZGVC6YAHXFYF4XYF75V7K4PJMEJEM53WHKPFI75Q, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.873475218 (195h53m40.889144821s) [noise=true , address=77.90.31.17:4770 , piece_id=EZVXBCBLVCAUY5SY6XGU4UCYQNAYEQXXJ4CSRPSVVIQOUIHHGGIA, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.874051549 (195h53m40.889721158s) [noise=true , address=145.79.227.8:4335 , piece_id=QKRCS2TPOVBEDW4IIEUG2T4XRGT7LV5TKZ46V7URCDJEGW4MKTXQ, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.874606386 (195h53m40.890275986s) [noise=true , address=145.79.227.38:4982 , piece_id=FEQDEN664AQUSUOIENGXRTKNIHFJSPNPVUMSO2FRO5VUBTDUW5TQ, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.875088289 (195h53m40.890757901s) [noise=true , address=145.79.227.3:4154 , piece_id=YTAPQYSW4TRFL5HR2FYFEWH5UJVE65A7YLEEHLDYA7QA3QTWGS3Q, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.875643352 (195h53m40.89131296s) [noise=true , address=145.79.227.43:4158 , piece_id=U7F5ISEB4TLASJOZZL2Y7YQH64GUQ25ZCRC66EAZ6DUPYKOOQUEQ, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.875757309 (195h53m40.891426917s) [noise=true , address=145.79.227.7:4296 , piece_id=PRKLW7JCHXZFG26FXSYQXCUYNVEP5IVVXPVONZEFWAUDOYHROQLA, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.876359538 (195h53m40.892029144s) [noise=true , address=45.91.116.10:4083 , piece_id=ICB2HQZVONJN5NNDBMIH42J5ZABWBG4HH4UMUCU7OI4XPMI2FO6A, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:37.876647362 (195h53m40.892316965s) [noise=true , address=77.90.31.14:4664 , piece_id=35FJ52TJJWAV52U5HUHG6Y7S237LMOQHFK5UBMDUAPYEME6NAYPA, seg_pos={0 0}, upload=188167]: Uploading piece...
Mar 18 20:04:38.481436417 (195h53m41.497106018s) [upload=188167]: Next returned nil segment
Mar 18 20:04:38.481453167 (195h53m41.497122768s) [upload=188167]: Waiting for error group managing segments...
Mar 18 20:04:38.616084096 (195h53m41.631753699s) [noise=true , address=45.91.116.9:4296 , piece_id=KIPFBZRWSRPNLSUORGP23GILTRDKOL45DDZ45ZD4GROHGLB6LGYQ, seg_pos={0 0}, upload=188167]: Done uploading piece. err: <nil>
Mar 18 20:04:38.628464308 (195h53m41.644133918s) [noise=true , address=45.91.116.13:4156 , piece_id=3RKQNBDJZ5VF6AFPMCPVKUUNPQXBIHQ6ULFKFF2JMOMGX6GI6LYA, seg_pos={0 0}, upload=188167]: Done uploading piece. err: <nil>
Mar 18 20:04:38.633353567 (195h53m41.649023167s) [noise=true , address=45.91.116.10:4083 , piece_id=ICB2HQZVONJN5NNDBMIH42J5ZABWBG4HH4UMUCU7OI4XPMI2FO6A, seg_pos={0 0}, upload=188167]: Done uploading piece. err: <nil>
Mar 18 20:04:38.639810638 (195h53m41.655480239s) [noise=true , address=45.91.116.6:4975 , piece_id=47T5WWY2MM4QFLZQGDFJXGE6OJUSE5RQSRAVSFJTZ6HC2J3LC5RA, seg_pos={0 0}, upload=188167]: Done uploading piece. err: <nil>
Mar 18 20:04:38.702713441 (195h53m41.718383045s) [noise=true , address=77.90.31.14:4664 , piece_id=35FJ52TJJWAV52U5HUHG6Y7S237LMOQHFK5UBMDUAPYEME6NAYPA, seg_pos={0 0}, upload=188167]: Done uploading piece. err: <nil>
Mar 18 20:04:38.706857177 (195h53m41.722526785s) [noise=true , address=77.90.31.17:4770 , piece_id=EZVXBCBLVCAUY5SY6XGU4UCYQNAYEQXXJ4CSRPSVVIQOUIHHGGIA, seg_pos={0 0}, upload=188167]: Done uploading piece. err: <nil>
Mar 18 20:04:38.729137396 (195h53m41.744807035s) [noise=true , address=79.127.219.38:4821 , piece_id=4IMEOEQEKC2SFBU7IMU6Y4R4YZKL4USZNCNCGC3YQGNWVFFLJCHQ, seg_pos={0 0}, upload=188167]: Done uploading piece. err: <nil>
Mar 18 20:05:05.130332565 (195h54m8.146002177s) [noise=true , address=145.79.227.8:4335 , piece_id=QKRCS2TPOVBEDW4IIEUG2T4XRGT7LV5TKZ46V7URCDJEGW4MKTXQ, seg_pos={0 0}, upload=188167]: Done uploading piece. err: ecclient: upload failed (node:1sKfG447ZZTQ6QJ1ohCxbfvm5Sv8u8azreBAqoC6EFbvXQdzwN, address:145.79.227.8:4335): protocol: rpc: dial tcp 145.79.227.8:4335: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:05.130662590 (195h54m8.146332203s) [noise=true , address=145.79.227.42:4122 , piece_id=Y5UPWO4HDUXPLLXJKIDZ7NGZXUPEMYNVLISS5ESRBKWZQ7E4UGBA, seg_pos={0 0}, upload=188167]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: rpc: dial tcp 145.79.227.42:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:05.131058934 (195h54m8.146728547s) [noise=true , address=145.79.227.38:4982 , piece_id=FEQDEN664AQUSUOIENGXRTKNIHFJSPNPVUMSO2FRO5VUBTDUW5TQ, seg_pos={0 0}, upload=188167]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: rpc: dial tcp 145.79.227.38:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:05.131360159 (195h54m8.147029769s) [noise=true , address=145.79.227.3:4154 , piece_id=YTAPQYSW4TRFL5HR2FYFEWH5UJVE65A7YLEEHLDYA7QA3QTWGS3Q, seg_pos={0 0}, upload=188167]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:145.79.227.3:4154): protocol: rpc: dial tcp 145.79.227.3:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:05.131962753 (195h54m8.147632365s) [noise=true , address=145.79.227.49:4342 , piece_id=RHFVXERXH4A2F4ZY6BUO2VZ224F4MBNEU6OFJ7LFSTK7LCQ2DQ5Q, seg_pos={0 0}, upload=188167]: Done uploading piece. err: ecclient: upload failed (node:1pYxAYKuf8jtEBtsH5cvx8hmTfqrMv5Z4c6taF9hVgRDmx8KoB, address:145.79.227.49:4342): protocol: rpc: dial tcp 145.79.227.49:4342: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:05.132249023 (195h54m8.147918634s) [noise=true , address=145.79.227.20:4440 , piece_id=CKC5YGI7GRI6IMZBYGQS3ZJ6L5TLMCGE5JELGZVLC6UQAWMJTA5A, seg_pos={0 0}, upload=188167]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:145.79.227.20:4440): protocol: rpc: dial tcp 145.79.227.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:05.132347151 (195h54m8.148016744s) [noise=true , address=145.79.227.43:4158 , piece_id=U7F5ISEB4TLASJOZZL2Y7YQH64GUQ25ZCRC66EAZ6DUPYKOOQUEQ, seg_pos={0 0}, upload=188167]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:145.79.227.43:4158): protocol: rpc: dial tcp 145.79.227.43:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:05.133010289 (195h54m8.148679901s) [noise=true , address=145.79.227.33:4870 , piece_id=CGO24G7TIG6FZGVC6YAHXFYF4XYF75V7K4PJMEJEM53WHKPFI75Q, seg_pos={0 0}, upload=188167]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: rpc: dial tcp 145.79.227.33:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:05.133507623 (195h54m8.149177266s) [noise=true , address=145.79.227.7:4296 , piece_id=PRKLW7JCHXZFG26FXSYQXCUYNVEP5IVVXPVONZEFWAUDOYHROQLA, seg_pos={0 0}, upload=188167]: Done uploading piece. err: ecclient: upload failed (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1, address:145.79.227.7:4296): protocol: rpc: dial tcp 145.79.227.7:4296: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:05:05.133636670 (195h54m8.149306272s) [seg_pos={0 0}, upload=188167]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil>
Mar 18 20:05:05.244519130 (195h54m8.260188737s) [upload=188167]: Done uploading segments...
Mar 18 20:05:05.244561183 (195h54m8.260230794s) [upload=188167]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7)
Duration: 30.11591823s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [..................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001997977s trace:
https://jaeger.dev.storj.io/trace/3a56f45eae3f3794
FAIL (2026-03-18 20:04:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002030998s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.......................X..X...X..X..X........X..X..X...X....] duration:
18.606650202s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:04:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 18.606707759s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [........................XXX.XXX...............X.............] duration:
16.98558359s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:04:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 16.985698747s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [...................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000989297s trace:
https://jaeger.dev.storj.io/trace/77d51659be8068b2
FAIL (2026-03-18 20:03:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001090505s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
37 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration:
30.118253859s trace:
https://jaeger.dev.storj.io/trace/4c2e6a9c6e50215c
FAIL (2026-03-18 20:03:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Mar 18 20:03:27.839589551 (195h52m30.855259152s) [upload=188148]: Starting upload
Mar 18 20:03:27.839763892 (195h52m30.855433493s) [upload=188148]: Done starting upload
Mar 18 20:03:27.839861528 (195h52m30.855531129s) [upload=188148]: Uploading segments...
Mar 18 20:03:27.840224383 (195h52m30.855893976s) [upload=188148]: Got next segment. Inline: false
Mar 18 20:03:27.840279120 (195h52m30.85594872s) [upload=188148]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:03:27.840284349 (195h52m30.85595395s) [upload=188148]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:03:30.735479971 (195h52m33.751149575s) [seg_pos={0 0}, upload=188148]: Begin upload segment...
Mar 18 20:03:30.735571298 (195h52m33.751240905s) [seg_pos={0 0}, upload=188148]: Done begin upload segment.
Mar 18 20:03:30.735628711 (195h52m33.75129832s) [noise=true , address=145.79.227.38:4980 , piece_id=NK7WAPA2N57UMJVA7JRLMML3P4I2AF7EEBI3IWO6MLQFTOCKXDZA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.735642268 (195h52m33.751311874s) [noise=true , address=45.91.116.6:4989 , piece_id=25FNNO3H3N5BE7NGU6X3KFNAFMPPD7JO56ZVDSVTN5NQNCJ7DEAA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.736217813 (195h52m33.751887416s) [noise=true , address=45.91.116.13:4154 , piece_id=AA4BHD2WHDZV4WKVA734FBZK36P4EKSKMRNXENTLXVFCNGAU62BA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.736228291 (195h52m33.751897901s) [noise=true , address=79.127.219.41:4930 , piece_id=DNVK6ILSAC3UZPSQMIG6DQCZ34WZO7UAP4JT4DX4XCIWCH3XWVCA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.736613053 (195h52m33.752282654s) [noise=true , address=145.79.227.30:4761 , piece_id=2TNEUZM3Z5KFOA4FSQZPJDDHWWIE6SENIRG6VL4LWR6LIAQFWIXQ, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.736673821 (195h52m33.752343421s) [noise=true , address=145.79.227.25:4623 , piece_id=TCXI5LP7Z4LWBUX3W7ISL4ZBVNWIFUQKEKU7NB54UBLPWIK7WQFA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.737030585 (195h52m33.752700196s) [noise=true , address=45.91.116.8:4049 , piece_id=FDQOV2HIYDOTOVJHMMLOZWETCQIBPV43URCVGPFRJHDD5T7XLWHA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.737114312 (195h52m33.752783915s) [noise=true , address=77.90.31.11:4558 , piece_id=IOQIFYSLKP5KHHRBCRXLAS57ENXCDSEGCJ3QDQRYJ36R7BS7I3SA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.737449815 (195h52m33.753119418s) [noise=true , address=77.90.31.13:4627 , piece_id=ZWY6I7F6YPM6TFLBQXSXMAAXNSEBQNQGPFMHTX4S5KDFNAQ2MZUQ, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.737940879 (195h52m33.753610481s) [noise=true , address=45.91.116.5:4299 , piece_id=XT4CIIQFNJGYZZMGUKQGIWUHXTR3L3CPDQ6IUDUABJN5JPXE2YKA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.738072751 (195h52m33.753742357s) [noise=true , address=79.127.219.46:4891 , piece_id=PENTV5YY7NAEVDS2LQWUGVSU7I2CJOYERMQT7ISWGNTCZ6JPYYPA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.738389353 (195h52m33.754058953s) [noise=true , address=145.79.227.52:4449 , piece_id=G7UKJ6R6TLW6VPX32XDJRBCHZJCYCLSAVHEYSFH6UKIBYP2RXGMA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.738421878 (195h52m33.754091479s) [noise=true , address=145.79.227.14:4840 , piece_id=2H3IYTLXHXD7LSRTCBDVUX4B5QXQBFE7T72CY6ZTXWYKMZGFNTUA, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.738898771 (195h52m33.754568371s) [noise=true , address=45.91.116.9:4264 , piece_id=C3LTAZDUK45JNI5WRHNWLHVVXPKI6SIH46X32C26EVSNQKWMRYOQ, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.739486973 (195h52m33.755156574s) [noise=true , address=77.90.31.16:4736 , piece_id=LHQG4K3YVQFTU7AGBKO53J4F5WJUY3WCSZL3HKMZ5WX7E6UXDWWQ, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:30.739868698 (195h52m33.755538292s) [noise=true , address=45.91.116.7:4011 , piece_id=KWEJTUD36G5KOUOI7COI34RSHH4LZU3IBRTGNEESO3AYTZDTWKEQ, seg_pos={0 0}, upload=188148]: Uploading piece...
Mar 18 20:03:31.335007213 (195h52m34.350676812s) [upload=188148]: Next returned nil segment
Mar 18 20:03:31.335025723 (195h52m34.350695346s) [upload=188148]: Waiting for error group managing segments...
Mar 18 20:03:31.471136004 (195h52m34.486805624s) [noise=true , address=77.90.31.11:4558 , piece_id=IOQIFYSLKP5KHHRBCRXLAS57ENXCDSEGCJ3QDQRYJ36R7BS7I3SA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.477186283 (195h52m34.492855884s) [noise=true , address=45.91.116.9:4264 , piece_id=C3LTAZDUK45JNI5WRHNWLHVVXPKI6SIH46X32C26EVSNQKWMRYOQ, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.482733591 (195h52m34.498403192s) [noise=true , address=45.91.116.5:4299 , piece_id=XT4CIIQFNJGYZZMGUKQGIWUHXTR3L3CPDQ6IUDUABJN5JPXE2YKA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.490935908 (195h52m34.50660552s) [noise=true , address=45.91.116.6:4989 , piece_id=25FNNO3H3N5BE7NGU6X3KFNAFMPPD7JO56ZVDSVTN5NQNCJ7DEAA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.494927937 (195h52m34.510597539s) [noise=true , address=45.91.116.13:4154 , piece_id=AA4BHD2WHDZV4WKVA734FBZK36P4EKSKMRNXENTLXVFCNGAU62BA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.495934730 (195h52m34.511604338s) [noise=true , address=45.91.116.7:4011 , piece_id=KWEJTUD36G5KOUOI7COI34RSHH4LZU3IBRTGNEESO3AYTZDTWKEQ, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.499640288 (195h52m34.515309899s) [noise=true , address=45.91.116.8:4049 , piece_id=FDQOV2HIYDOTOVJHMMLOZWETCQIBPV43URCVGPFRJHDD5T7XLWHA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.532841094 (195h52m34.548510701s) [noise=true , address=79.127.219.46:4891 , piece_id=PENTV5YY7NAEVDS2LQWUGVSU7I2CJOYERMQT7ISWGNTCZ6JPYYPA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.548728323 (195h52m34.564397931s) [noise=true , address=77.90.31.13:4627 , piece_id=ZWY6I7F6YPM6TFLBQXSXMAAXNSEBQNQGPFMHTX4S5KDFNAQ2MZUQ, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.569841212 (195h52m34.585510834s) [noise=true , address=77.90.31.16:4736 , piece_id=LHQG4K3YVQFTU7AGBKO53J4F5WJUY3WCSZL3HKMZ5WX7E6UXDWWQ, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:31.600099510 (195h52m34.615769112s) [noise=true , address=79.127.219.41:4930 , piece_id=DNVK6ILSAC3UZPSQMIG6DQCZ34WZO7UAP4JT4DX4XCIWCH3XWVCA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: <nil>
Mar 18 20:03:57.534255307 (195h53m0.54992491s) [noise=true , address=145.79.227.38:4980 , piece_id=NK7WAPA2N57UMJVA7JRLMML3P4I2AF7EEBI3IWO6MLQFTOCKXDZA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: ecclient: upload failed (node:1hzDqFsTcWDF5LLW9tm77hLADjUkhbLqkE3DBSHyJf3PyiHszu, address:145.79.227.38:4980): protocol: rpc: dial tcp 145.79.227.38:4980: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:03:57.534391324 (195h53m0.550060942s) [noise=true , address=145.79.227.25:4623 , piece_id=TCXI5LP7Z4LWBUX3W7ISL4ZBVNWIFUQKEKU7NB54UBLPWIK7WQFA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: ecclient: upload failed (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5, address:145.79.227.25:4623): protocol: rpc: dial tcp 145.79.227.25:4623: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:03:57.534804504 (195h53m0.550474118s) [noise=true , address=145.79.227.52:4449 , piece_id=G7UKJ6R6TLW6VPX32XDJRBCHZJCYCLSAVHEYSFH6UKIBYP2RXGMA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: ecclient: upload failed (node:12f14pHDEEamEnHmeZCe38rjaE6EYWYkGMPrZMsbSDmApN5rckq, address:145.79.227.52:4449): protocol: rpc: dial tcp 145.79.227.52:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:03:57.535123725 (195h53m0.550793312s) [noise=true , address=145.79.227.14:4840 , piece_id=2H3IYTLXHXD7LSRTCBDVUX4B5QXQBFE7T72CY6ZTXWYKMZGFNTUA, seg_pos={0 0}, upload=188148]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: rpc: dial tcp 145.79.227.14:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:03:57.535161507 (195h53m0.550831098s) [noise=true , address=145.79.227.30:4761 , piece_id=2TNEUZM3Z5KFOA4FSQZPJDDHWWIE6SENIRG6VL4LWR6LIAQFWIXQ, seg_pos={0 0}, upload=188148]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: rpc: dial tcp 145.79.227.30:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:03:57.535198271 (195h53m0.550867857s) [seg_pos={0 0}, upload=188148]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Mar 18 20:03:57.649897204 (195h53m0.665566819s) [upload=188148]: Done uploading segments...
Mar 18 20:03:57.649982515 (195h53m0.665652128s) [upload=188148]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.118280328s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [........................X..X...X..X..X........X..X..X...X...] duration:
15.302095322s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:03:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 15.302125229s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [....................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001226775s trace:
https://jaeger.dev.storj.io/trace/22c5638b5e6d1bfc
FAIL (2026-03-18 20:03:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001260235s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.........................XXX.XXX...............X............] duration:
16.387380581s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:03:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 16.387407208s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [.........................X..X...X..X..X........X..X..X...X..] duration:
13.924781994s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:03:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 13.924813385s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
36 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration:
30.115112468s trace:
https://jaeger.dev.storj.io/trace/6924e6cdf3289d6b
FAIL (2026-03-18 20:02:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
929s) [upload=188131]: Starting upload
Mar 18 20:02:24.970912554 (195h51m27.986582148s) [upload=188131]: Done starting upload
Mar 18 20:02:24.971046143 (195h51m27.986715738s) [upload=188131]: Uploading segments...
Mar 18 20:02:24.971431763 (195h51m27.987101354s) [upload=188131]: Got next segment. Inline: false
Mar 18 20:02:24.971501554 (195h51m27.987171156s) [upload=188131]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:02:24.971507291 (195h51m27.987176892s) [upload=188131]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:02:27.859271893 (195h51m30.874941499s) [seg_pos={0 0}, upload=188131]: Begin upload segment...
Mar 18 20:02:27.859344367 (195h51m30.875013955s) [seg_pos={0 0}, upload=188131]: Done begin upload segment.
Mar 18 20:02:27.859365775 (195h51m30.875035383s) [noise=true , address=145.79.227.36:4521 , piece_id=UVMS7FPTK3YNEM7NWV5MDF26R2KQFILJGT4TZVFPU2ZS5BWCDNAQ, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.859381329 (195h51m30.87505094s) [noise=true , address=79.127.219.38:4818 , piece_id=DHSVUM7CJPRZEAPRLKBJCH3NB3OMTFL2QP6L2DAKMQXWFTHGG6HA, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.859453684 (195h51m30.875123294s) [noise=true , address=45.91.116.14:4189 , piece_id=3ZU7A7RBYPGM2FRH45ZSXEDZUVCQ2MQ3OPE2ZGXG53SQGPJKSNYQ, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.860535796 (195h51m30.876205403s) [noise=true , address=145.79.227.13:4805 , piece_id=QLTYRYJGKITSAK32O2W6CIYZMFVW26WIBCHO35HL2IO7TY5OM4OQ, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.861034241 (195h51m30.876703846s) [noise=true , address=145.79.227.5:4221 , piece_id=QHH6744AVCTROPUHWS5ZZS355IFEIMLOJBHERRYZBLXITVVJ722Q, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.861445052 (195h51m30.877114653s) [noise=true , address=77.90.31.19:4846 , piece_id=2UC4CFG4TFUK6IJUNYOLD2KHWZCNGHT5IPHFHIEYQ267XQDEODZA, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.862137966 (195h51m30.877807578s) [noise=true , address=145.79.227.16:4915 , piece_id=KNH32HHQFPAD3MP6BVQH62V5VQIEKJ6AJFDSW7NO57XZZEYSWO4A, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.862712511 (195h51m30.878382362s) [noise=true , address=145.79.227.11:4734 , piece_id=BV5SOULHHHWJFGJ4CEOKR3BH5AV62ENYARGCLVHOL2IYASET6DCA, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.863374246 (195h51m30.87904385s) [noise=true , address=145.79.227.20:4441 , piece_id=NCIYEG64XY6OUTFENQL24UDIHY5DNZMABMZVRHRPVSVZBVFGF53Q, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.864006764 (195h51m30.879676368s) [noise=true , address=77.90.31.12:4590 , piece_id=3WPIIMLCNDZS5Z6ZRHJIFBK7DBZ6SUMRYVLMJQELP3SPAIV3AHRQ, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.864613335 (195h51m30.880282925s) [noise=true , address=45.91.116.10:4084 , piece_id=6532LXJKOVPV46NCYLS22SS3QH6V473ZXZMX2XOADXSQGGXTR2QQ, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.865263441 (195h51m30.880933042s) [noise=true , address=77.90.31.21:4882 , piece_id=VR4D35424K6C2KCBXRRU4VZBUVAKNFAVGNUK7JJONERBS4AI37NQ, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.866124521 (195h51m30.881794133s) [noise=true , address=45.91.116.8:4046 , piece_id=LMX2L6SY4ZXF66YBNI24SHDWQV6FJRY4JVHYOOWLUOX4ITVLQQJA, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.871593077 (195h51m30.887262688s) [noise=true , address=145.79.227.38:4982 , piece_id=P5FYRFEKHISU6OHP4GY7IDVDD6WYPTHHCJBSELNPVM6AM6YES52Q, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.875384142 (195h51m30.891053754s) [noise=true , address=45.91.116.9:4296 , piece_id=2HB662ZVLSMBFAAJGTRVP2NIY3642RM42QIVZ5FUTHTAARM3CXBA, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:27.879692933 (195h51m30.895362538s) [noise=true , address=145.79.227.46:4604 , piece_id=RPMH2LNB36L3CUCMEAMYPZOKG3N62XH4XDB2ZTHAJ57FETHFDS7Q, seg_pos={0 0}, upload=188131]: Uploading piece...
Mar 18 20:02:28.488919858 (195h51m31.50458946s) [upload=188131]: Next returned nil segment
Mar 18 20:02:28.488937555 (195h51m31.504607162s) [upload=188131]: Waiting for error group managing segments...
Mar 18 20:02:28.626591488 (195h51m31.642261083s) [noise=true , address=77.90.31.21:4882 , piece_id=VR4D35424K6C2KCBXRRU4VZBUVAKNFAVGNUK7JJONERBS4AI37NQ, seg_pos={0 0}, upload=188131]: Done uploading piece. err: <nil>
Mar 18 20:02:28.629756013 (195h51m31.645425614s) [noise=true , address=45.91.116.9:4296 , piece_id=2HB662ZVLSMBFAAJGTRVP2NIY3642RM42QIVZ5FUTHTAARM3CXBA, seg_pos={0 0}, upload=188131]: Done uploading piece. err: <nil>
Mar 18 20:02:28.630305449 (195h51m31.645975053s) [noise=true , address=45.91.116.8:4046 , piece_id=LMX2L6SY4ZXF66YBNI24SHDWQV6FJRY4JVHYOOWLUOX4ITVLQQJA, seg_pos={0 0}, upload=188131]: Done uploading piece. err: <nil>
Mar 18 20:02:28.632535062 (195h51m31.648204667s) [noise=true , address=45.91.116.14:4189 , piece_id=3ZU7A7RBYPGM2FRH45ZSXEDZUVCQ2MQ3OPE2ZGXG53SQGPJKSNYQ, seg_pos={0 0}, upload=188131]: Done uploading piece. err: <nil>
Mar 18 20:02:28.634151053 (195h51m31.64982066s) [noise=true , address=45.91.116.10:4084 , piece_id=6532LXJKOVPV46NCYLS22SS3QH6V473ZXZMX2XOADXSQGGXTR2QQ, seg_pos={0 0}, upload=188131]: Done uploading piece. err: <nil>
Mar 18 20:02:28.634590390 (195h51m31.650259994s) [noise=true , address=77.90.31.19:4846 , piece_id=2UC4CFG4TFUK6IJUNYOLD2KHWZCNGHT5IPHFHIEYQ267XQDEODZA, seg_pos={0 0}, upload=188131]: Done uploading piece. err: <nil>
Mar 18 20:02:28.635075934 (195h51m31.650745537s) [noise=true , address=77.90.31.12:4590 , piece_id=3WPIIMLCNDZS5Z6ZRHJIFBK7DBZ6SUMRYVLMJQELP3SPAIV3AHRQ, seg_pos={0 0}, upload=188131]: Done uploading piece. err: <nil>
Mar 18 20:02:28.665518174 (195h51m31.681187775s) [noise=true , address=79.127.219.38:4818 , piece_id=DHSVUM7CJPRZEAPRLKBJCH3NB3OMTFL2QP6L2DAKMQXWFTHGG6HA, seg_pos={0 0}, upload=188131]: Done uploading piece. err: <nil>
Mar 18 20:02:54.668326479 (195h51m57.683996089s) [noise=true , address=145.79.227.46:4604 , piece_id=RPMH2LNB36L3CUCMEAMYPZOKG3N62XH4XDB2ZTHAJ57FETHFDS7Q, seg_pos={0 0}, upload=188131]: Done uploading piece. err: ecclient: upload failed (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy, address:145.79.227.46:4604): protocol: rpc: dial tcp 145.79.227.46:4604: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:54.668620946 (195h51m57.684290558s) [noise=true , address=145.79.227.13:4805 , piece_id=QLTYRYJGKITSAK32O2W6CIYZMFVW26WIBCHO35HL2IO7TY5OM4OQ, seg_pos={0 0}, upload=188131]: Done uploading piece. err: ecclient: upload failed (node:12YEVsEwZwfWsnZFyaSkd9Dn53Tb9EFUyWEVDwTQeMJkcjwrj7x, address:145.79.227.13:4805): protocol: rpc: dial tcp 145.79.227.13:4805: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:54.668722902 (195h51m57.684392511s) [noise=true , address=145.79.227.11:4734 , piece_id=BV5SOULHHHWJFGJ4CEOKR3BH5AV62ENYARGCLVHOL2IYASET6DCA, seg_pos={0 0}, upload=188131]: Done uploading piece. err: ecclient: upload failed (node:12GLb5LQje9wwivnj8Wj1tXcnS6mFb2Bco5saNwgmFJBVDujrv7, address:145.79.227.11:4734): protocol: rpc: dial tcp 145.79.227.11:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:54.668892333 (195h51m57.684561941s) [noise=true , address=145.79.227.5:4221 , piece_id=QHH6744AVCTROPUHWS5ZZS355IFEIMLOJBHERRYZBLXITVVJ722Q, seg_pos={0 0}, upload=188131]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:145.79.227.5:4221): protocol: rpc: dial tcp 145.79.227.5:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:54.669117627 (195h51m57.684787236s) [noise=true , address=145.79.227.36:4521 , piece_id=UVMS7FPTK3YNEM7NWV5MDF26R2KQFILJGT4TZVFPU2ZS5BWCDNAQ, seg_pos={0 0}, upload=188131]: Done uploading piece. err: ecclient: upload failed (node:12GQHySYUuqERoQyP4fCib2nRMNJhgpP2NkDzVxz6nurLkeJ5VD, address:145.79.227.36:4521): protocol: rpc: dial tcp 145.79.227.36:4521: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:54.669445098 (195h51m57.685114708s) [noise=true , address=145.79.227.20:4441 , piece_id=NCIYEG64XY6OUTFENQL24UDIHY5DNZMABMZVRHRPVSVZBVFGF53Q, seg_pos={0 0}, upload=188131]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: rpc: dial tcp 145.79.227.20:4441: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:54.669863437 (195h51m57.685533024s) [noise=true , address=145.79.227.16:4915 , piece_id=KNH32HHQFPAD3MP6BVQH62V5VQIEKJ6AJFDSW7NO57XZZEYSWO4A, seg_pos={0 0}, upload=188131]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: rpc: dial tcp 145.79.227.16:4915: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:54.669948689 (195h51m57.685618294s) [noise=true , address=145.79.227.38:4982 , piece_id=P5FYRFEKHISU6OHP4GY7IDVDD6WYPTHHCJBSELNPVM6AM6YES52Q, seg_pos={0 0}, upload=188131]: Done uploading piece. err: ecclient: upload failed (node:1NxoYJXCXmTvQncbkXt6yH7mhEBBrzVdzhgL5J6W5TD4N3pkaG, address:145.79.227.38:4982): protocol: rpc: dial tcp 145.79.227.38:4982: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:54.669986177 (195h51m57.685655766s) [seg_pos={0 0}, upload=188131]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Mar 18 20:02:54.782036721 (195h51m57.797706335s) [upload=188131]: Done uploading segments...
Mar 18 20:02:54.782088387 (195h51m57.797757997s) [upload=188131]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.115139829s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [.....................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001509282s trace:
https://jaeger.dev.storj.io/trace/28e36359cc02353d
FAIL (2026-03-18 20:02:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001535652s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [..........................XXX.XXX...............X...........] duration:
14.326549404s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:02:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.326577662s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [..........................X..X...X..X..X........X..X..X...X.] duration:
15.894316662s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:02:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 15.894347429s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
35 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration:
30.119170692s trace:
https://jaeger.dev.storj.io/trace/31f0e1e347003c99
FAIL (2026-03-18 20:02:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
Mar 18 20:01:32.612298809 (195h50m35.627968416s) [upload=188116]: Starting upload
Mar 18 20:01:32.612481044 (195h50m35.628150658s) [upload=188116]: Done starting upload
Mar 18 20:01:32.613016248 (195h50m35.62868585s) [upload=188116]: Uploading segments...
Mar 18 20:01:32.613108820 (195h50m35.628778421s) [upload=188116]: Got next segment. Inline: false
Mar 18 20:01:32.613206695 (195h50m35.62887631s) [upload=188116]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:01:32.613234782 (195h50m35.62890439s) [upload=188116]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:35.935220957 (195h50m38.950890557s) [seg_pos={0 0}, upload=188116]: Begin upload segment...
Mar 18 20:01:35.935297180 (195h50m38.950966771s) [seg_pos={0 0}, upload=188116]: Done begin upload segment.
Mar 18 20:01:35.935333228 (195h50m38.951002829s) [noise=true , address=145.79.227.46:4604 , piece_id=PZD2YI7LCB5YNMJTW262Z33WKXYX42F3NNOD3KVNJCK4RSP25MSQ, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.935415264 (195h50m38.95108487s) [noise=true , address=79.127.219.45:4853 , piece_id=TIY7F2XZMOYWNMPY6NPQEX42GIT7MNIHXRADRKNJ2XLCN6R4ZLMA, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.936358875 (195h50m38.952028481s) [noise=true , address=145.79.227.40:4051 , piece_id=IWYG76ILBFPJFVAVRL5UUR4V7SO7JFM5WRBXDVC7J3TY3FWQR77Q, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.937081177 (195h50m38.952750786s) [noise=true , address=45.91.116.7:4019 , piece_id=6KYHGVDCBNCICZIVCNHMMEV6R2RIH6EPRIHDYB67WVSRWRMDXYKQ, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.937214572 (195h50m38.952884179s) [noise=true , address=45.91.116.6:4974 , piece_id=AP7PWADNRNYZGBBAB7ICGNA33FGRRPZAGCKNOQOBGTU25FBJAHIA, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.938009078 (195h50m38.953678685s) [noise=true , address=145.79.227.32:4833 , piece_id=4VB4PHZBABVQWTFG3IOG5KAJHA3JKJLBG44JMRLXBVIUQ5YIUISA, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.938192105 (195h50m38.953861697s) [noise=true , address=45.91.116.13:4188 , piece_id=LFTOLEDCR5WNGWE2O4EEYWSM4RQQBTJRHKVTR3PS2FYFSPSOMHQQ, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.938646776 (195h50m38.954316388s) [noise=true , address=77.90.31.11:4557 , piece_id=OL3BB7TOAO3LITXICLUWBYO36NEIUDUJ4OC6LWMIY62UIVIOU4MA, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.938879447 (195h50m38.95454905s) [noise=true , address=77.90.31.17:4770 , piece_id=N4UHNINNF4SWPKKI6F6Z3RDDHAOYWOFXZHKOGF2GKQY4Q4PKUCGA, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.939235897 (195h50m38.954905503s) [noise=true , address=145.79.227.42:4122 , piece_id=NK54ZDU6IWBBGXMIJSRLVGHMSZW2X3EIJDVLP2LMUMUOAODZOJ5A, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.939566839 (195h50m38.95523644s) [noise=true , address=45.91.116.4:4941 , piece_id=DX3NWIEVLI2RCMGGE6FEZEAXFLQZOTJGZ3ZFXUVHXCFTH7F7FZYA, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.939758083 (195h50m38.955427686s) [noise=true , address=45.91.116.8:4079 , piece_id=BXPZ5GCKWC6OU4HBLDR5NBDTKEUNPC7IYY7NWFZRHQORRMHC2LWQ, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.940220199 (195h50m38.955889812s) [noise=true , address=77.90.31.16:4738 , piece_id=TFRFW22YXBFPFZP2R6XZ3TTLP6FQQ5D7P3ZBJLL7EGBN4OBFI3YQ, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.940299555 (195h50m38.955969163s) [noise=true , address=45.91.116.14:4192 , piece_id=4WVVHJKVG2NUEQCY4YUOKR6PNQXAAVAAGZCPATSFWU6KDVXJN2IQ, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.940823160 (195h50m38.95649276s) [noise=true , address=145.79.227.52:4450 , piece_id=AKFHJ565WJRAF5F5L5RNIUTUVRZE7EMWHPR4V6C4AN2WRI357Z3A, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:35.940996260 (195h50m38.956665871s) [noise=true , address=145.79.227.47:4265 , piece_id=3VMGPJCSYHLEHTG227JSKAYLCBC6CCSICXGJ3VHWUCEXKT6V7HNA, seg_pos={0 0}, upload=188116]: Uploading piece...
Mar 18 20:01:36.544363334 (195h50m39.560032928s) [upload=188116]: Next returned nil segment
Mar 18 20:01:36.544377492 (195h50m39.560047092s) [upload=188116]: Waiting for error group managing segments...
Mar 18 20:01:36.682587384 (195h50m39.698256988s) [noise=true , address=45.91.116.4:4941 , piece_id=DX3NWIEVLI2RCMGGE6FEZEAXFLQZOTJGZ3ZFXUVHXCFTH7F7FZYA, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:01:36.682871885 (195h50m39.698541488s) [noise=true , address=45.91.116.7:4019 , piece_id=6KYHGVDCBNCICZIVCNHMMEV6R2RIH6EPRIHDYB67WVSRWRMDXYKQ, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:01:36.683498861 (195h50m39.699168473s) [noise=true , address=77.90.31.11:4557 , piece_id=OL3BB7TOAO3LITXICLUWBYO36NEIUDUJ4OC6LWMIY62UIVIOU4MA, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:01:36.683793368 (195h50m39.699462972s) [noise=true , address=45.91.116.14:4192 , piece_id=4WVVHJKVG2NUEQCY4YUOKR6PNQXAAVAAGZCPATSFWU6KDVXJN2IQ, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:01:36.684169857 (195h50m39.699839464s) [noise=true , address=45.91.116.8:4079 , piece_id=BXPZ5GCKWC6OU4HBLDR5NBDTKEUNPC7IYY7NWFZRHQORRMHC2LWQ, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:01:36.703982951 (195h50m39.719652541s) [noise=true , address=45.91.116.13:4188 , piece_id=LFTOLEDCR5WNGWE2O4EEYWSM4RQQBTJRHKVTR3PS2FYFSPSOMHQQ, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:01:36.752107266 (195h50m39.767776867s) [noise=true , address=77.90.31.17:4770 , piece_id=N4UHNINNF4SWPKKI6F6Z3RDDHAOYWOFXZHKOGF2GKQY4Q4PKUCGA, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:01:36.754323195 (195h50m39.769992795s) [noise=true , address=79.127.219.45:4853 , piece_id=TIY7F2XZMOYWNMPY6NPQEX42GIT7MNIHXRADRKNJ2XLCN6R4ZLMA, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:01:36.755473503 (195h50m39.771143111s) [noise=true , address=45.91.116.6:4974 , piece_id=AP7PWADNRNYZGBBAB7ICGNA33FGRRPZAGCKNOQOBGTU25FBJAHIA, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:01:36.772342268 (195h50m39.788011879s) [noise=true , address=77.90.31.16:4738 , piece_id=TFRFW22YXBFPFZP2R6XZ3TTLP6FQQ5D7P3ZBJLL7EGBN4OBFI3YQ, seg_pos={0 0}, upload=188116]: Done uploading piece. err: <nil>
Mar 18 20:02:02.306957602 (195h51m5.322627217s) [noise=true , address=145.79.227.42:4122 , piece_id=NK54ZDU6IWBBGXMIJSRLVGHMSZW2X3EIJDVLP2LMUMUOAODZOJ5A, seg_pos={0 0}, upload=188116]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: rpc: dial tcp 145.79.227.42:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:02.307004499 (195h51m5.322674112s) [noise=true , address=145.79.227.32:4833 , piece_id=4VB4PHZBABVQWTFG3IOG5KAJHA3JKJLBG44JMRLXBVIUQ5YIUISA, seg_pos={0 0}, upload=188116]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: rpc: dial tcp 145.79.227.32:4833: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:02.307487886 (195h51m5.323157501s) [noise=true , address=145.79.227.52:4450 , piece_id=AKFHJ565WJRAF5F5L5RNIUTUVRZE7EMWHPR4V6C4AN2WRI357Z3A, seg_pos={0 0}, upload=188116]: Done uploading piece. err: ecclient: upload failed (node:12XE5kZtVU6FeUR5NF2mhwypKiJahTUyt4qtRbo3PUJp6T5sdms, address:145.79.227.52:4450): protocol: rpc: dial tcp 145.79.227.52:4450: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:02:02.307748133 (195h51m5.323417746s) [noise=true , address=145.79.227.40:4051 , piece_id=IWYG76ILBFPJFVAVRL5UUR4V7SO7JFM5WRBXDVC7J3TY3FWQR77Q, seg_pos={0 0}, upload=188116]: Done uploading piece. err: ecclient: upload failed (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n, address:145.79.227.40:4051): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:02:02.308002262 (195h51m5.323671869s) [noise=true , address=145.79.227.47:4265 , piece_id=3VMGPJCSYHLEHTG227JSKAYLCBC6CCSICXGJ3VHWUCEXKT6V7HNA, seg_pos={0 0}, upload=188116]: Done uploading piece. err: ecclient: upload failed (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ, address:145.79.227.47:4265): protocol: rpc: dial tcp 145.79.227.47:4265: i/o timeout; context deadline exceeded
Mar 18 20:02:02.308380818 (195h51m5.32405043s) [noise=true , address=145.79.227.46:4604 , piece_id=PZD2YI7LCB5YNMJTW262Z33WKXYX42F3NNOD3KVNJCK4RSP25MSQ, seg_pos={0 0}, upload=188116]: Done uploading piece. err: ecclient: upload failed (node:129d1v3mjqB3FHC2VmbY9Sih6C5fRDrV3kGLiv9NfkDc7JovXjy, address:145.79.227.46:4604): protocol: rpc: dial tcp 145.79.227.46:4604: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Mar 18 20:02:02.308432230 (195h51m5.324101832s) [seg_pos={0 0}, upload=188116]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Mar 18 20:02:02.423505408 (195h51m5.439175013s) [upload=188116]: Done uploading segments...
Mar 18 20:02:02.423614955 (195h51m5.439284563s) [upload=188116]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 30.119189935s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [...........................XXX.XXX...............X..........] duration:
16.773276658s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:02:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 16.773301406s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [......................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001006566s trace:
https://jaeger.dev.storj.io/trace/1bc0cd03ceb8c2f0
FAIL (2026-03-18 20:01:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001037798s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [...........................X..X...X..X..X........X..X..X...X] duration:
1m0.129249021s trace:
https://jaeger.dev.storj.io/trace/7f59026f88f99097
FAIL (2026-03-18 20:01:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
ad=188105]: Uploading segments...
Mar 18 20:00:50.229949662 (195h49m53.245619265s) [upload=188105]: Got next segment. Inline: false
Mar 18 20:00:50.230055981 (195h49m53.245725589s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:50.230062239 (195h49m53.24573184s) [upload=188105]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:53.477059332 (195h49m56.492728932s) [seg_pos={0 0}, upload=188105]: Begin upload segment...
Mar 18 20:00:53.477164841 (195h49m56.492834435s) [seg_pos={0 0}, upload=188105]: Done begin upload segment.
Mar 18 20:00:53.477197075 (195h49m56.492866682s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477218439 (195h49m56.49288804s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.477852309 (195h49m56.493521919s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478011870 (195h49m56.493681476s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478329586 (195h49m56.493999188s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478341794 (195h49m56.494011394s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478624071 (195h49m56.494293671s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478778710 (195h49m56.494448302s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.478912853 (195h49m56.494582456s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479145486 (195h49m56.494815081s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479201225 (195h49m56.494870829s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479493876 (195h49m56.495163479s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479839896 (195h49m56.495509501s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.479963425 (195h49m56.495633017s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480239518 (195h49m56.495909125s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.480694017 (195h49m56.496363626s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481226433 (195h49m56.496896041s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:53.481733251 (195h49m56.497402851s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Uploading piece...
Mar 18 20:00:54.029641675 (195h49m57.045311284s) [upload=188105]: Next returned nil segment
Mar 18 20:00:54.029660427 (195h49m57.045330044s) [upload=188105]: Waiting for error group managing segments...
Mar 18 20:00:54.177546162 (195h49m57.193215764s) [noise=true , address=77.90.31.10:4526 , piece_id=LFLDUQ6DRIMBSEGHV3QN5AFXC66PZNLHFV3HJZDU2KSZSHQNK33Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.191925384 (195h49m57.207594992s) [noise=true , address=77.90.31.14:4683 , piece_id=WMWLOAPJU5TX2IUMFIHBBEA7RAHR3PIYGTHWKNEREPZZXHB5JHQQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.198200690 (195h49m57.213870295s) [noise=true , address=45.91.116.14:4218 , piece_id=TB6NCEIKMZAK64XS3UMHFEMTPWVFVDCDQNT6RMDSVBC6OFQPIYFQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.251863256 (195h49m57.267532864s) [noise=true , address=79.127.219.45:4881 , piece_id=AWE2ZJIP4KZNF4DIGA7CH3RI5QBRL2KKOQGYL6LKL2QD6FXKS7ZA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.384419843 (195h49m57.400089446s) [noise=true , address=79.127.219.38:4830 , piece_id=2H5KT53224MH7N3XAJ25ZJQRRZNAOUNCWN4PN7KHEU3JECQVENDA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.563992445 (195h49m57.579662046s) [noise=true , address=167.17.72.69:4731 , piece_id=UELRA6HUF5MZEMHZAUWITBHMCTI35ZJS5QHIPAWDDQIBPN7MB36Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.594229581 (195h49m57.609899208s) [noise=true , address=167.17.72.71:4801 , piece_id=TVCGOIOFT3PUVW5JTVUKB7AZICQ6YSMQ5Y42YYZSHS6QRENWTNFA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.728073041 (195h49m57.743742644s) [noise=true , address=193.149.164.69:4078 , piece_id=RUVEKUB3HEZVSDMM6J4RGDVGSIJTZPM4BFRTVILY2IAYER7JUDLQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.730721899 (195h49m57.746391501s) [noise=true , address=193.149.164.70:4136 , piece_id=PO36TUAD3K7XJMHGXUIRCRERM6JLGGYARTRTS5G7ABM5Q5WUQNMQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.741202718 (195h49m57.756872329s) [noise=true , address=193.149.164.71:4315 , piece_id=3FUPXQ4BNS4X5RB2WLQMHEYZVD4CLK27JG66XPAWUGQ36KCDLBUA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.771477179 (195h49m57.78714678s) [noise=true , address=79.127.205.235:4252 , piece_id=TJP444URLC7KQHR44Z5GG4QIHFSCIS3FRUY2KHC6J7FOZX2EMS3Q, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:54.829071811 (195h49m57.844741431s) [noise=true , address=79.127.205.234:4204 , piece_id=JEPQA2CB2L53WFXRWFMADKI6BVLPQXUYWOKQK7DJRNJXWSUNKOIQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:00:55.391451427 (195h49m58.407121036s) [noise=true , address=79.127.205.233:4163 , piece_id=S2AE2Q5J7B2PJKJKMDG2EW5NXPVUYCAC5ERG64SCTHLFFHBTRFPQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: <nil>
Mar 18 20:01:49.923788818 (195h50m52.939458427s) [noise=true , address=145.79.227.23:4580 , piece_id=NHPNTRUUQQXWYZHINYCIB3CYWWBAKL4YB3IJOJHJS6ZNRB2Q7OOQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12569SWj8WmR8tWoBXk7XJK1KS6BZMcYHrQjPKGfhj21xFPQ1W3, address:145.79.227.23:4580): protocol: rpc: dial tcp 145.79.227.23:4580: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.923804605 (195h50m52.939474211s) [noise=true , address=145.79.227.7:4313 , piece_id=XPQYB7S2QXIP6Z6JO5HPFEACHEXV65TT5OTDNUOYEXQK3ZLNUZBA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:145.79.227.7:4313): protocol: rpc: dial tcp 145.79.227.7:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924076787 (195h50m52.939746397s) [noise=true , address=145.79.227.18:4017 , piece_id=JRGGUXS3EYGLKGCSBN6QCMKXCNQDZWXUSIIIU3LOOJMQ6ZCKMJVQ, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12wS9YpUnsjgkdTPSowozp8txmqAhfF4n8s7JVpLb2tdRgQeLw8, address:145.79.227.18:4017): protocol: rpc: dial tcp 145.79.227.18:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924103627 (195h50m52.939773233s) [noise=true , address=145.79.227.43:4191 , piece_id=WMLZKWFSEIZLFGFCICXLLQ7PHANI4YNOSV5DKIG3I2IZ3O2YDWGA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: rpc: dial tcp 145.79.227.43:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924279185 (195h50m52.939948794s) [noise=true , address=145.79.227.22:4517 , piece_id=BHJP6TLHQUHGWE3LWXHJKSTOKX2TFPMECMS3SDAY3S3ORDCNVOPA, seg_pos={0 0}, upload=188105]: Done uploading piece. err: ecclient: upload failed (node:1wSJWmZaz4FNEiar3UGW17tJc6XvaW4erEXbpAsWnE6FvuLRzT, address:145.79.227.22:4517): protocol: rpc: dial tcp 145.79.227.22:4517: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:49.924309946 (195h50m52.939979546s) [seg_pos={0 0}, upload=188105]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 20:01:50.050765493 (195h50m53.066435097s) [upload=188105]: Done uploading segments...
Mar 18 20:01:50.050802707 (195h50m53.066472307s) [upload=188105]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 1m0.12933215s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration:
30.117454047s trace:
https://jaeger.dev.storj.io/trace/20cd7dde16b7b551
FAIL (2026-03-18 20:01:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 6)
======LOGS======
load=188108]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:01:02.357893380 (195h50m5.373562986s) [upload=188108]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:01:05.626701584 (195h50m8.642371208s) [seg_pos={0 0}, upload=188108]: Begin upload segment...
Mar 18 20:01:05.626763792 (195h50m8.64243339s) [seg_pos={0 0}, upload=188108]: Done begin upload segment.
Mar 18 20:01:05.626804452 (195h50m8.64247406s) [noise=true , address=45.91.116.4:4955 , piece_id=JPGTXBEANYWSNB7XWUU7AJQGTA57PLPIMTQ4TVQZ6OUZTEOLIHOQ, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.626848670 (195h50m8.642518281s) [noise=true , address=145.79.227.6:4260 , piece_id=DHHNTFEQENIPPOEYROWRKPYCNHA4KFGOEIPE4YEA6LNHXAR6CBHA, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.627653073 (195h50m8.643322685s) [noise=true , address=145.79.227.11:4733 , piece_id=35MAVHUDOZ5K2DMAR5BNJ3QF46CGQKKUPYMPHQWBE3DKVGVUMJCQ, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.628130134 (195h50m8.643799748s) [noise=true , address=145.79.227.49:4338 , piece_id=SZOT7BYJ7Q4JJ6QZCSBO5I5FUE34POOZI5V4A2CAAU2RJ2O54CNQ, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.629370266 (195h50m8.645039927s) [noise=true , address=77.90.31.19:4841 , piece_id=SM2HOKNYNWTSV7IBSM62ULTTNDNRCJCCVGUT5QXK2I3772ISH4TA, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.629446752 (195h50m8.645116372s) [noise=true , address=145.79.227.36:4520 , piece_id=UMYCOGJBLNR7VGI4UOBMWSQL4P5IQGG3FO44K7RGINPP3C2SWLZA, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.629542721 (195h50m8.645212332s) [noise=true , address=145.79.227.5:4225 , piece_id=WU4QORCDPDG35Z6364DPW3TRGKYRIMBVVQUGX3JTE6USTU752NDA, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.631130290 (195h50m8.646799902s) [noise=true , address=77.90.31.12:4594 , piece_id=CPYGHE7KUN2AJVW3NPOT2GETFGIAQKYP5THP5EJPQYYDNJFFIYBQ, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.631191289 (195h50m8.646860901s) [noise=true , address=145.79.227.39:4017 , piece_id=ZDSXGW4U4UB2UKVLHDN7HS3TPWVXJDT6PELPPO3L5XN2GRAOMPYA, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.632579520 (195h50m8.648249132s) [noise=true , address=45.91.116.8:4070 , piece_id=JAA7N5TEGWK4OZH3G7BFZ54UD4M37GYESHLPPL3G7SLBZXXMS5XA, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.633967184 (195h50m8.649636797s) [noise=true , address=145.79.227.12:4770 , piece_id=IPZL5IWTOVTPH3UOOPSQSSCDBDEASC2X6UXRBZIORN3PUSRCRMLA, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.636032520 (195h50m8.651702133s) [noise=true , address=145.79.227.46:4603 , piece_id=TYFRKDHD755G4E6XALFPVLZ4MYPMCUW77GSDIU3FJ2UJCIMX5BXA, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.636494288 (195h50m8.652163899s) [noise=true , address=79.127.219.46:4893 , piece_id=ULR3MBCYRCS3U7VMKUOBWFSCPR5BVT7LDNFYHERIXRYCI7AUOUWQ, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.637300934 (195h50m8.652970549s) [noise=true , address=145.79.227.51:4413 , piece_id=TX4P5GEA5KU2VYUMQ6M4ZEPDXRXDK5G33GYSRIQXLKAO56VUDHFQ, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.638983218 (195h50m8.654652827s) [noise=true , address=77.90.31.10:4518 , piece_id=HBZQWF4SCEZALKSJLQF64PCJ3DSFZJUEPFFVLBP7NEOGPX4TDDQQ, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:05.639847925 (195h50m8.655517537s) [noise=true , address=145.79.227.19:4081 , piece_id=GHQQ2LX7ABHTMZ4XIWHKITNYS35QCIEQCWNT7PN63BP42S645HQQ, seg_pos={0 0}, upload=188108]: Uploading piece...
Mar 18 20:01:06.234719147 (195h50m9.250388758s) [upload=188108]: Next returned nil segment
Mar 18 20:01:06.234743000 (195h50m9.250412611s) [upload=188108]: Waiting for error group managing segments...
Mar 18 20:01:06.367898864 (195h50m9.383568468s) [noise=true , address=77.90.31.12:4594 , piece_id=CPYGHE7KUN2AJVW3NPOT2GETFGIAQKYP5THP5EJPQYYDNJFFIYBQ, seg_pos={0 0}, upload=188108]: Done uploading piece. err: <nil>
Mar 18 20:01:06.377466174 (195h50m9.393135798s) [noise=true , address=77.90.31.10:4518 , piece_id=HBZQWF4SCEZALKSJLQF64PCJ3DSFZJUEPFFVLBP7NEOGPX4TDDQQ, seg_pos={0 0}, upload=188108]: Done uploading piece. err: <nil>
Mar 18 20:01:06.378585633 (195h50m9.394255246s) [noise=true , address=45.91.116.4:4955 , piece_id=JPGTXBEANYWSNB7XWUU7AJQGTA57PLPIMTQ4TVQZ6OUZTEOLIHOQ, seg_pos={0 0}, upload=188108]: Done uploading piece. err: <nil>
Mar 18 20:01:06.378914162 (195h50m9.394583768s) [noise=true , address=77.90.31.19:4841 , piece_id=SM2HOKNYNWTSV7IBSM62ULTTNDNRCJCCVGUT5QXK2I3772ISH4TA, seg_pos={0 0}, upload=188108]: Done uploading piece. err: <nil>
Mar 18 20:01:06.386246977 (195h50m9.401916578s) [noise=true , address=45.91.116.8:4070 , piece_id=JAA7N5TEGWK4OZH3G7BFZ54UD4M37GYESHLPPL3G7SLBZXXMS5XA, seg_pos={0 0}, upload=188108]: Done uploading piece. err: <nil>
Mar 18 20:01:06.386275094 (195h50m9.401944696s) [noise=true , address=79.127.219.46:4893 , piece_id=ULR3MBCYRCS3U7VMKUOBWFSCPR5BVT7LDNFYHERIXRYCI7AUOUWQ, seg_pos={0 0}, upload=188108]: Done uploading piece. err: <nil>
Mar 18 20:01:32.054067627 (195h50m35.069737236s) [noise=true , address=145.79.227.46:4603 , piece_id=TYFRKDHD755G4E6XALFPVLZ4MYPMCUW77GSDIU3FJ2UJCIMX5BXA, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX, address:145.79.227.46:4603): protocol: rpc: dial tcp 145.79.227.46:4603: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:32.054516957 (195h50m35.070186564s) [noise=true , address=145.79.227.11:4733 , piece_id=35MAVHUDOZ5K2DMAR5BNJ3QF46CGQKKUPYMPHQWBE3DKVGVUMJCQ, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:145.79.227.11:4733): protocol: rpc: dial tcp 145.79.227.11:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:32.054566603 (195h50m35.070236215s) [noise=true , address=145.79.227.51:4413 , piece_id=TX4P5GEA5KU2VYUMQ6M4ZEPDXRXDK5G33GYSRIQXLKAO56VUDHFQ, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:1oMVt1KZoNSDX5jKHBXw4WSr69eQYxcY3Z1nxLBsHr19Btw7Gy, address:145.79.227.51:4413): protocol: rpc: dial tcp 145.79.227.51:4413: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:32.054712109 (195h50m35.070381711s) [noise=true , address=145.79.227.39:4017 , piece_id=ZDSXGW4U4UB2UKVLHDN7HS3TPWVXJDT6PELPPO3L5XN2GRAOMPYA, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:12ocB4w2b2owjyWVCcq59Ymo2prJfSFmK5YogQDjGBroF11Tegk, address:145.79.227.39:4017): protocol: rpc: dial tcp 145.79.227.39:4017: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:32.054954746 (195h50m35.070624359s) [noise=true , address=145.79.227.49:4338 , piece_id=SZOT7BYJ7Q4JJ6QZCSBO5I5FUE34POOZI5V4A2CAAU2RJ2O54CNQ, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: rpc: dial tcp 145.79.227.49:4338: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:32.055110428 (195h50m35.070780033s) [noise=true , address=145.79.227.36:4520 , piece_id=UMYCOGJBLNR7VGI4UOBMWSQL4P5IQGG3FO44K7RGINPP3C2SWLZA, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:145.79.227.36:4520): protocol: rpc: dial tcp 145.79.227.36:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:32.055204718 (195h50m35.07087433s) [noise=true , address=145.79.227.6:4260 , piece_id=DHHNTFEQENIPPOEYROWRKPYCNHA4KFGOEIPE4YEA6LNHXAR6CBHA, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:1aLsgrV3k6vUSavgPkPGA2FF9HR3ryN9bnHutruFJHW1GYixeL, address:145.79.227.6:4260): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:32.055344852 (195h50m35.071014457s) [noise=true , address=145.79.227.5:4225 , piece_id=WU4QORCDPDG35Z6364DPW3TRGKYRIMBVVQUGX3JTE6USTU752NDA, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:12QtY3yN6J1H8htRneePHu4RmbN7A1xkgywphMDVsVxhTczGupY, address:145.79.227.5:4225): protocol: rpc: dial tcp 145.79.227.5:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:32.055692074 (195h50m35.071361692s) [noise=true , address=145.79.227.12:4770 , piece_id=IPZL5IWTOVTPH3UOOPSQSSCDBDEASC2X6UXRBZIORN3PUSRCRMLA, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:32.055990035 (195h50m35.071659653s) [noise=true , address=145.79.227.19:4081 , piece_id=GHQQ2LX7ABHTMZ4XIWHKITNYS35QCIEQCWNT7PN63BP42S645HQQ, seg_pos={0 0}, upload=188108]: Done uploading piece. err: ecclient: upload failed (node:12t9rGfCQ7bLkNnZzaPVgkE2Y9rXsiFp1XGdeECzv9M1jk8Qqmp, address:145.79.227.19:4081): protocol: rpc: dial tcp 145.79.227.19:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:32.056046885 (195h50m35.071716479s) [seg_pos={0 0}, upload=188108]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil>
Mar 18 20:01:32.170044002 (195h50m35.185713602s) [upload=188108]: Done uploading segments...
Mar 18 20:01:32.170083279 (195h50m35.185752879s) [upload=188108]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6)
Duration: 30.117473077s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXX] duration:
30.122205888s trace:
https://jaeger.dev.storj.io/trace/4d0be4326c51162a
FAIL (2026-03-18 20:01:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 9)
======LOGS======
Mar 18 20:00:32.089541738 (195h49m35.105211349s) [upload=188100]: Starting upload
Mar 18 20:00:32.089869778 (195h49m35.105539367s) [upload=188100]: Done starting upload
Mar 18 20:00:32.090230743 (195h49m35.105900334s) [upload=188100]: Uploading segments...
Mar 18 20:00:32.090440243 (195h49m35.106109835s) [upload=188100]: Got next segment. Inline: false
Mar 18 20:00:32.090505373 (195h49m35.106174973s) [upload=188100]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 20:00:32.090511041 (195h49m35.106180641s) [upload=188100]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 20:00:35.348518542 (195h49m38.364188152s) [seg_pos={0 0}, upload=188100]: Begin upload segment...
Mar 18 20:00:35.348622542 (195h49m38.364292134s) [seg_pos={0 0}, upload=188100]: Done begin upload segment.
Mar 18 20:00:35.348682274 (195h49m38.364351874s) [noise=true , address=145.79.227.19:4082 , piece_id=LAHKWBCY7APADJ6KHD4YYPAKLNAIYW2UM6C5TP7NT2YDVAEK6VEQ, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.350133869 (195h49m38.365803478s) [noise=true , address=145.79.227.34:4906 , piece_id=CEGLQURNW66HNPK6N3TT7KEWBFOWZYIHVCWH66MMRPVNUWLYQCUQ, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.350170158 (195h49m38.365839767s) [noise=true , address=145.79.227.41:4090 , piece_id=76I3SJGMRW4GKFZTFCNZUFHRDKOGKKLBNWIXHXFXYXGQREU4QLAQ, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.350270238 (195h49m38.365939849s) [noise=true , address=145.79.227.2:4117 , piece_id=EIADLGVMK2F357FU3UYEAIH5SOK5UW6BRTTZ4FMFM7DTL2VV5IWQ, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.350748406 (195h49m38.36641801s) [noise=true , address=45.91.116.5:4297 , piece_id=ZTJV6RCNFTZ5K7E3G2PTRY344VUDZSF5ZFJRV66VCZOTEFZI6LCA, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.350761440 (195h49m38.366431052s) [noise=true , address=77.90.31.21:4880 , piece_id=H3Z5XSJT7VGTF6XDE4M4B2RR6ODQIIZJABMGXPRBHCSEYCJPSNSA, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.351210440 (195h49m38.366880051s) [noise=true , address=45.91.116.14:4191 , piece_id=R5UQRD7ST4VZ2WSLHIT47FVRFBZ6OCAYKTKV7EGUTYG2TIRWAYHQ, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.351226356 (195h49m38.366895967s) [noise=true , address=145.79.227.48:4301 , piece_id=266BITZUFREAXPGTJASJZWR5VFFVYELJMMRVIIRPYOTV632K67SA, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.351694076 (195h49m38.367363687s) [noise=true , address=77.90.31.12:4590 , piece_id=IXLCVQ3LTBHE7MDIJT5RFB67PKQHWNR6GBB46QRTJ5MFD3I2KYUA, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.352135292 (195h49m38.367804885s) [noise=true , address=145.79.227.21:4473 , piece_id=KSH6GGRB7PRQYMA3W3J7Q6TBZK6JCN5WDRJ2B5HMHL6ZGQRHEBVA, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.352151840 (195h49m38.367821442s) [noise=true , address=45.91.116.12:4260 , piece_id=YYIWLWX7KZPMIJUTRWHHQDBOH7F7Y6TXRV5LWK5NGSY2V4UE4JSA, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.352190802 (195h49m38.367860389s) [noise=true , address=145.79.227.37:4565 , piece_id=DFDWHUZTYFK3DBIDSVKEHDARASTU5LNS25WDMFPAFMVCCSB3PTEQ, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.352624981 (195h49m38.368294592s) [noise=true , address=45.91.116.11:4152 , piece_id=VC5P3U6X3PBRFTTWCF55HVWHGMMSQDWKHUG4YWEBQ6FU7ZLMDJVQ, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.353109855 (195h49m38.368779465s) [noise=true , address=77.90.31.19:4844 , piece_id=2VP2A73NS2WUYICMELDYNXOULSHVL5XF4UNLNKGTPPB6FORD6G2A, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.353601606 (195h49m38.369271224s) [noise=true , address=77.90.31.18:4809 , piece_id=HLKIHC5FD5FLO2KZTDINA6JIOH54NPTYDRRHS7UZJY23VABFLMNA, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.354492419 (195h49m38.37016201s) [noise=true , address=45.91.116.9:4295 , piece_id=7PYLEIKMAOVPPM7HIEWXFECQ66YRRNYTMFXR5ISLJIYGLPUIFGKQ, seg_pos={0 0}, upload=188100]: Uploading piece...
Mar 18 20:00:35.953103232 (195h49m38.968772841s) [upload=188100]: Next returned nil segment
Mar 18 20:00:35.953120455 (195h49m38.968790061s) [upload=188100]: Waiting for error group managing segments...
Mar 18 20:00:36.086505965 (195h49m39.102175569s) [noise=true , address=77.90.31.21:4880 , piece_id=H3Z5XSJT7VGTF6XDE4M4B2RR6ODQIIZJABMGXPRBHCSEYCJPSNSA, seg_pos={0 0}, upload=188100]: Done uploading piece. err: <nil>
Mar 18 20:00:36.089507089 (195h49m39.10517669s) [noise=true , address=45.91.116.5:4297 , piece_id=ZTJV6RCNFTZ5K7E3G2PTRY344VUDZSF5ZFJRV66VCZOTEFZI6LCA, seg_pos={0 0}, upload=188100]: Done uploading piece. err: <nil>
Mar 18 20:00:36.092046349 (195h49m39.107715957s) [noise=true , address=77.90.31.18:4809 , piece_id=HLKIHC5FD5FLO2KZTDINA6JIOH54NPTYDRRHS7UZJY23VABFLMNA, seg_pos={0 0}, upload=188100]: Done uploading piece. err: <nil>
Mar 18 20:00:36.101399530 (195h49m39.117069134s) [noise=true , address=45.91.116.12:4260 , piece_id=YYIWLWX7KZPMIJUTRWHHQDBOH7F7Y6TXRV5LWK5NGSY2V4UE4JSA, seg_pos={0 0}, upload=188100]: Done uploading piece. err: <nil>
Mar 18 20:00:36.105683735 (195h49m39.121353341s) [noise=true , address=45.91.116.9:4295 , piece_id=7PYLEIKMAOVPPM7HIEWXFECQ66YRRNYTMFXR5ISLJIYGLPUIFGKQ, seg_pos={0 0}, upload=188100]: Done uploading piece. err: <nil>
Mar 18 20:00:36.105893109 (195h49m39.121562714s) [noise=true , address=45.91.116.11:4152 , piece_id=VC5P3U6X3PBRFTTWCF55HVWHGMMSQDWKHUG4YWEBQ6FU7ZLMDJVQ, seg_pos={0 0}, upload=188100]: Done uploading piece. err: <nil>
Mar 18 20:00:36.106525668 (195h49m39.122195272s) [noise=true , address=77.90.31.12:4590 , piece_id=IXLCVQ3LTBHE7MDIJT5RFB67PKQHWNR6GBB46QRTJ5MFD3I2KYUA, seg_pos={0 0}, upload=188100]: Done uploading piece. err: <nil>
Mar 18 20:00:36.110752819 (195h49m39.126422422s) [noise=true , address=45.91.116.14:4191 , piece_id=R5UQRD7ST4VZ2WSLHIT47FVRFBZ6OCAYKTKV7EGUTYG2TIRWAYHQ, seg_pos={0 0}, upload=188100]: Done uploading piece. err: <nil>
Mar 18 20:00:36.116442066 (195h49m39.132111674s) [noise=true , address=77.90.31.19:4844 , piece_id=2VP2A73NS2WUYICMELDYNXOULSHVL5XF4UNLNKGTPPB6FORD6G2A, seg_pos={0 0}, upload=188100]: Done uploading piece. err: <nil>
Mar 18 20:01:01.782737847 (195h50m4.798407461s) [noise=true , address=145.79.227.34:4906 , piece_id=CEGLQURNW66HNPK6N3TT7KEWBFOWZYIHVCWH66MMRPVNUWLYQCUQ, seg_pos={0 0}, upload=188100]: Done uploading piece. err: ecclient: upload failed (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw, address:145.79.227.34:4906): protocol: rpc: dial tcp 145.79.227.34:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:01.783138371 (195h50m4.798807983s) [noise=true , address=145.79.227.41:4090 , piece_id=76I3SJGMRW4GKFZTFCNZUFHRDKOGKKLBNWIXHXFXYXGQREU4QLAQ, seg_pos={0 0}, upload=188100]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:145.79.227.41:4090): protocol: rpc: dial tcp 145.79.227.41:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 20:01:01.783756990 (195h50m4.799426603s) [noise=true , address=145.79.227.19:4082 , piece_id=LAHKWBCY7APADJ6KHD4YYPAKLNAIYW2UM6C5TP7NT2YDVAEK6VEQ, seg_pos={0 0}, upload=188100]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:145.79.227.19:4082): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:01.784067314 (195h50m4.799736932s) [noise=true , address=145.79.227.37:4565 , piece_id=DFDWHUZTYFK3DBIDSVKEHDARASTU5LNS25WDMFPAFMVCCSB3PTEQ, seg_pos={0 0}, upload=188100]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:01.784330128 (195h50m4.799999743s) [noise=true , address=145.79.227.48:4301 , piece_id=266BITZUFREAXPGTJASJZWR5VFFVYELJMMRVIIRPYOTV632K67SA, seg_pos={0 0}, upload=188100]: Done uploading piece. err: ecclient: upload failed (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, address:145.79.227.48:4301): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:01.784496426 (195h50m4.800166039s) [noise=true , address=145.79.227.21:4473 , piece_id=KSH6GGRB7PRQYMA3W3J7Q6TBZK6JCN5WDRJ2B5HMHL6ZGQRHEBVA, seg_pos={0 0}, upload=188100]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:01.784583727 (195h50m4.800253348s) [noise=true , address=145.79.227.2:4117 , piece_id=EIADLGVMK2F357FU3UYEAIH5SOK5UW6BRTTZ4FMFM7DTL2VV5IWQ, seg_pos={0 0}, upload=188100]: Done uploading piece. err: ecclient: upload failed (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the, address:145.79.227.2:4117): protocol: context deadline exceeded; context deadline exceeded
Mar 18 20:01:01.784633180 (195h50m4.800302789s) [seg_pos={0 0}, upload=188100]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil>
Mar 18 20:01:01.896274366 (195h50m4.911943971s) [upload=188100]: Done uploading segments...
Mar 18 20:01:01.896354336 (195h50m4.912023946s) [upload=188100]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9)
Duration: 30.122232326s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
32 out of 60 is failed (threshold=6) [.......................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXXX] duration:
5.001802693s trace:
https://jaeger.dev.storj.io/trace/525acb0454fcd8b4
FAIL (2026-03-18 20:00:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001828854s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [............................XXX.XXX...............X.........] duration:
17.038447668s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 20:00:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.038522723s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
31 out of 60 is failed (threshold=6) [........................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXXX] duration:
5.000873092s trace:
https://jaeger.dev.storj.io/trace/297e3d66c18666af
FAIL (2026-03-18 20:00:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000901773s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [............................X..X...X..X..X........X..X..X...] duration:
17.870673575s trace:
https://jaeger.dev.storj.io/trace/657fdcaa051c01a8
FAIL (2026-03-18 19:59:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
18 19:56:59.292408464 (195h46m2.308078058s) [upload=188049]: Uploading segments...
Mar 18 19:56:59.292755979 (195h46m2.30842558s) [upload=188049]: Got next segment. Inline: false
Mar 18 19:56:59.292808462 (195h46m2.308478061s) [upload=188049]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:56:59.292813871 (195h46m2.308483471s) [upload=188049]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:57:02.876720251 (195h46m5.892389863s) [seg_pos={0 0}, upload=188049]: Begin upload segment...
Mar 18 19:57:02.876793971 (195h46m5.892463581s) [seg_pos={0 0}, upload=188049]: Done begin upload segment.
Mar 18 19:57:02.876833263 (195h46m5.892502873s) [noise=true , address=45.91.116.5:4313 , piece_id=NBWJEW5YG75SETFUT5GDB7SY7AV2FKSXDVYU6GDLH3LDBJWZIEXQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.876890974 (195h46m5.892560585s) [noise=true , address=45.91.116.8:4061 , piece_id=RVNNCRXRDGITJ4ANLAZWW5BIQQV6XZPA3664Z6YK4VYZ6WWFUGZQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.877956175 (195h46m5.893625785s) [noise=true , address=45.91.116.12:4252 , piece_id=3S6PWGR6TJLKHO4QNMWD2QC5POZFB7LDIMO2IWLQ6WMZQHYPGUGA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.878298073 (195h46m5.89396768s) [noise=true , address=167.17.72.70:4755 , piece_id=WXEJWO6LCSL4XPKNR3K2UQDD4REU6VDGXTSONMIXC2W5OHZD5FLA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.879712704 (195h46m5.895382315s) [noise=true , address=5.231.73.4:4023 , piece_id=ZG6PQBPOBCODENWWP2AQ3VWKT5JJ6DYYXET7EV4WOA6XLSNDUDJQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.880345930 (195h46m5.896015539s) [noise=true , address=79.127.205.235:4261 , piece_id=MMBAEATMIDZI3PIPWTANIPEFY6LWLD36WSRS4DGVP7GNO2THGDJA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.880748175 (195h46m5.896417775s) [noise=true , address=193.149.164.71:4315 , piece_id=AFPIVEIWVLXPXAVIAMBYEUCTE2IV4UMV72UCB632TB4WYS5PK56A, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.881290748 (195h46m5.896960358s) [noise=true , address=193.149.164.66:4991 , piece_id=JFQCEGUEYXYJ7RWJ7W2U73HGVNO2LFRLZ4YUKKSGXYQGGCCOHHYA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.882022155 (195h46m5.897691756s) [noise=true , address=77.90.31.19:4875 , piece_id=OPWPGCXK3DPV7WHTCY57VN5NWM3HXWSSZBOKNSYVY2LEHCRU665A, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.882473710 (195h46m5.898143318s) [noise=true , address=45.91.116.6:4006 , piece_id=DK5KUCW3YZIV5ATUEXO3C4O7GEPSLMTPVGZFZAGXIUF3TSBGW4TQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.883003620 (195h46m5.898673224s) [noise=true , address=145.79.227.28:4749 , piece_id=RAUWPP47L2SQHEPUVNTRZIWFPG7BNDD5DDYSUGI63K7IGI6QS7AA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.883504542 (195h46m5.899174149s) [noise=true , address=167.17.72.69:4738 , piece_id=Y5UBOMFHBSJEX7QLHTR3BSP5OVS7J4HQNUP7TPKPJHSU43JKWTWQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.884006474 (195h46m5.899676078s) [noise=true , address=145.79.227.18:4039 , piece_id=DWKTIPR4SKDEH43DTFIPQZGRVPMYNXGAH4FMAY2USNNTUU2WIHPA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.884343043 (195h46m5.900012643s) [noise=true , address=145.79.227.36:4538 , piece_id=XK7DMW633XPQX7RUL4JKHD75BDQOVVBYJ5SBCAU6X4GWWF4DZ5BA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.884660141 (195h46m5.900329745s) [noise=true , address=145.79.227.49:4343 , piece_id=EUYXULIELHYJGX6UKMMOIWTOUBS6YCGEPIXSMINDVCUBBRUBL34A, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.885052951 (195h46m5.900722557s) [noise=true , address=145.79.227.32:4846 , piece_id=PXUAYXKNTDYZUBQLWUPG5D6K2L4LT7N73BXRXEDARWC36NICFJOQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.886175686 (195h46m5.901845308s) [noise=true , address=193.149.164.69:4073 , piece_id=2WMFLFXQ6TMOLKFU3NXT5CCLE5GN5IZR2JQO2ZXA7MCMZWQOWLKA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.886594862 (195h46m5.902264463s) [noise=true , address=193.149.164.70:4133 , piece_id=G54QWDQDBWQKGBQFLWK6NT26CJIQFWEC4MCWSWTKYE3FMHR6FMRA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:03.444101668 (195h46m6.459771278s) [upload=188049]: Next returned nil segment
Mar 18 19:57:03.444124472 (195h46m6.459794082s) [upload=188049]: Waiting for error group managing segments...
Mar 18 19:57:03.590092875 (195h46m6.605762476s) [noise=true , address=45.91.116.12:4252 , piece_id=3S6PWGR6TJLKHO4QNMWD2QC5POZFB7LDIMO2IWLQ6WMZQHYPGUGA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.603331674 (195h46m6.619001274s) [noise=true , address=45.91.116.6:4006 , piece_id=DK5KUCW3YZIV5ATUEXO3C4O7GEPSLMTPVGZFZAGXIUF3TSBGW4TQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.613170926 (195h46m6.628840529s) [noise=true , address=45.91.116.5:4313 , piece_id=NBWJEW5YG75SETFUT5GDB7SY7AV2FKSXDVYU6GDLH3LDBJWZIEXQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.617370351 (195h46m6.633039952s) [noise=true , address=45.91.116.8:4061 , piece_id=RVNNCRXRDGITJ4ANLAZWW5BIQQV6XZPA3664Z6YK4VYZ6WWFUGZQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.687019561 (195h46m6.702689162s) [noise=true , address=77.90.31.19:4875 , piece_id=OPWPGCXK3DPV7WHTCY57VN5NWM3HXWSSZBOKNSYVY2LEHCRU665A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.967066275 (195h46m6.982735877s) [noise=true , address=167.17.72.70:4755 , piece_id=WXEJWO6LCSL4XPKNR3K2UQDD4REU6VDGXTSONMIXC2W5OHZD5FLA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.977678776 (195h46m6.993348378s) [noise=true , address=5.231.73.4:4023 , piece_id=ZG6PQBPOBCODENWWP2AQ3VWKT5JJ6DYYXET7EV4WOA6XLSNDUDJQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.986006395 (195h46m7.001676s) [noise=true , address=167.17.72.69:4738 , piece_id=Y5UBOMFHBSJEX7QLHTR3BSP5OVS7J4HQNUP7TPKPJHSU43JKWTWQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.105385143 (195h46m7.121054743s) [noise=true , address=193.149.164.70:4133 , piece_id=G54QWDQDBWQKGBQFLWK6NT26CJIQFWEC4MCWSWTKYE3FMHR6FMRA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.113300230 (195h46m7.12896983s) [noise=true , address=193.149.164.71:4315 , piece_id=AFPIVEIWVLXPXAVIAMBYEUCTE2IV4UMV72UCB632TB4WYS5PK56A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.156743026 (195h46m7.172412637s) [noise=true , address=193.149.164.66:4991 , piece_id=JFQCEGUEYXYJ7RWJ7W2U73HGVNO2LFRLZ4YUKKSGXYQGGCCOHHYA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.212855060 (195h46m7.22852466s) [noise=true , address=79.127.205.235:4261 , piece_id=MMBAEATMIDZI3PIPWTANIPEFY6LWLD36WSRS4DGVP7GNO2THGDJA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.223915835 (195h46m7.239585447s) [noise=true , address=193.149.164.69:4073 , piece_id=2WMFLFXQ6TMOLKFU3NXT5CCLE5GN5IZR2JQO2ZXA7MCMZWQOWLKA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:59.001554644 (195h47m2.017224275s) [noise=true , address=145.79.227.49:4343 , piece_id=EUYXULIELHYJGX6UKMMOIWTOUBS6YCGEPIXSMINDVCUBBRUBL34A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.001715114 (195h47m2.017384752s) [noise=true , address=145.79.227.32:4846 , piece_id=PXUAYXKNTDYZUBQLWUPG5D6K2L4LT7N73BXRXEDARWC36NICFJOQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:1X6WKD6kuW1CgatniwYHDWrT9tfdq9sxB25UxAUBzTRXsJPyfH, address:145.79.227.32:4846): protocol: rpc: dial tcp 145.79.227.32:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.002131998 (195h47m2.017801609s) [noise=true , address=145.79.227.18:4039 , piece_id=DWKTIPR4SKDEH43DTFIPQZGRVPMYNXGAH4FMAY2USNNTUU2WIHPA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: rpc: dial tcp 145.79.227.18:4039: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.002164582 (195h47m2.017834216s) [noise=true , address=145.79.227.36:4538 , piece_id=XK7DMW633XPQX7RUL4JKHD75BDQOVVBYJ5SBCAU6X4GWWF4DZ5BA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:145.79.227.36:4538): protocol: rpc: dial tcp 145.79.227.36:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.002428976 (195h47m2.018098586s) [noise=true , address=145.79.227.28:4749 , piece_id=RAUWPP47L2SQHEPUVNTRZIWFPG7BNDD5DDYSUGI63K7IGI6QS7AA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:1CKpmdJXXJVjy5PYy8N82gquvqGPaaLE3Qxz2pTAbTUPnpBZCi, address:145.79.227.28:4749): protocol: rpc: dial tcp 145.79.227.28:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.002458729 (195h47m2.01812833s) [seg_pos={0 0}, upload=188049]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:57:59.118230032 (195h47m2.133899643s) [upload=188049]: Done uploading segments...
Mar 18 19:57:59.118301627 (195h47m2.133971217s) [upload=188049]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 17.870700558s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.............................XXX.XXX...............X........] duration:
15.992058949s trace:
https://jaeger.dev.storj.io/trace/15e13da99079fd26
FAIL (2026-03-18 19:59:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.992155725s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXX] duration:
30.124136341s trace:
https://jaeger.dev.storj.io/trace/1c7b26c0ac264b2
FAIL (2026-03-18 19:59:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
Mar 18 19:59:23.309433115 (195h48m26.325102721s) [upload=188081]: Starting upload
Mar 18 19:59:23.309489741 (195h48m26.325159349s) [upload=188081]: Done starting upload
Mar 18 19:59:23.309705006 (195h48m26.325374613s) [upload=188081]: Uploading segments...
Mar 18 19:59:23.309816528 (195h48m26.325486178s) [upload=188081]: Got next segment. Inline: false
Mar 18 19:59:23.309885541 (195h48m26.325555149s) [upload=188081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:59:23.309893507 (195h48m26.325563114s) [upload=188081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:59:26.437140826 (195h48m29.452810435s) [seg_pos={0 0}, upload=188081]: Begin upload segment...
Mar 18 19:59:26.437219627 (195h48m29.452889234s) [seg_pos={0 0}, upload=188081]: Done begin upload segment.
Mar 18 19:59:26.437268174 (195h48m29.452937783s) [noise=true , address=79.127.219.41:4925 , piece_id=CR6GCZU3E6Z6DSUMNN6AQGJEMPP6NJ5KT7BQHDLTKKYDCK4NYPWA, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.437285368 (195h48m29.452954977s) [noise=true , address=145.79.227.14:4842 , piece_id=NV3IH3AVYGUROSSTZ54HKNS3LSACCR3TJVM2FDQOFOLTX6MQTWYQ, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.437992986 (195h48m29.453662599s) [noise=true , address=45.91.116.7:4011 , piece_id=4Q5LRA5RL5RLR4XQ6RFZIBXJDJMWEK5SPUD5TBQIMZ337GH7PT6Q, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.438354998 (195h48m29.454024585s) [noise=true , address=45.91.116.9:4263 , piece_id=NZWJQOKC6RQHYPWOFBKYYZ73ZF2C4KQXMEMT56F6ZR64L23SR4TQ, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.438540169 (195h48m29.454209779s) [noise=true , address=77.90.31.15:4700 , piece_id=PKZJFZCDED5EEU2JER2O4O572KTXWMU7VYS2E476VDM73T5N3JWQ, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.439017310 (195h48m29.454686917s) [noise=true , address=77.90.31.19:4845 , piece_id=KMKLMDE62H67YUD74ZVSWD4OZ6CTHPCBBNXOQP7XIZROKO75RLGA, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.439619363 (195h48m29.45528897s) [noise=true , address=45.91.116.10:4088 , piece_id=SXJBIWC73Q6GV2U45PV6N3GIJH5NX5ZEJ35TFBGMTUZ4H23LCOOQ, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.440027913 (195h48m29.455697504s) [noise=true , address=77.90.31.13:4625 , piece_id=HYSJIJPCVIEN3UHIW4MS4SJONXQX4GLZBRAQD3M5LV7EFGFZEZJA, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.440247767 (195h48m29.455917377s) [noise=true , address=145.79.227.48:4303 , piece_id=CBSK3ULMM5OB5W5UYEXNOJBMGL3AN5NFKUTOUV4OGO74MQGY7Y2Q, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.440396953 (195h48m29.456066562s) [noise=true , address=145.79.227.47:4265 , piece_id=F7BSRBCLSISCSICRQYSN4CB7DTN6N77XXBGKLSURTCHUG4COOVUA, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.440726974 (195h48m29.456396592s) [noise=true , address=145.79.227.16:4913 , piece_id=6VA3TDQRU2AFSPF54A2QTYB633V5UATUH4EVGCDKDVRMYB7ZXEJA, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.441215855 (195h48m29.456885465s) [noise=true , address=145.79.227.33:4872 , piece_id=TBCLYWZ7PPKQTRMPFRIWQKE7NLHBDWPBDS2EFQ3KEBZQDW45HNXA, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.441472821 (195h48m29.457142411s) [noise=true , address=145.79.227.7:4298 , piece_id=N46DXUO65PDTHCVC4PZ37LBKN3JCTCTXCSIUT4ZNS3OBZNUEKXQQ, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.441959124 (195h48m29.457628724s) [noise=true , address=145.79.227.10:4404 , piece_id=OM3WIFEMP37FY53GTCMAXS6ZEICTWCSVOBH26UFTPICIUAZNULFQ, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.442346527 (195h48m29.458016128s) [noise=true , address=79.127.219.38:4817 , piece_id=JE4QG7HWYDJGXZPUDW7BJSERCUBOUJGCQAKU6RLC3URPGOKCINAA, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:26.442512457 (195h48m29.458182069s) [noise=true , address=145.79.227.13:4807 , piece_id=MYR24VBLJPY223FQKXGEBDUCF7NC47VLBIWCA5SMB4JCHKWMDELQ, seg_pos={0 0}, upload=188081]: Uploading piece...
Mar 18 19:59:27.032589887 (195h48m30.04825949s) [upload=188081]: Next returned nil segment
Mar 18 19:59:27.032614734 (195h48m30.048284334s) [upload=188081]: Waiting for error group managing segments...
Mar 18 19:59:27.165944745 (195h48m30.181614345s) [noise=true , address=77.90.31.19:4845 , piece_id=KMKLMDE62H67YUD74ZVSWD4OZ6CTHPCBBNXOQP7XIZROKO75RLGA, seg_pos={0 0}, upload=188081]: Done uploading piece. err: <nil>
Mar 18 19:59:27.173110006 (195h48m30.18877961s) [noise=true , address=45.91.116.10:4088 , piece_id=SXJBIWC73Q6GV2U45PV6N3GIJH5NX5ZEJ35TFBGMTUZ4H23LCOOQ, seg_pos={0 0}, upload=188081]: Done uploading piece. err: <nil>
Mar 18 19:59:27.181858283 (195h48m30.197527884s) [noise=true , address=79.127.219.41:4925 , piece_id=CR6GCZU3E6Z6DSUMNN6AQGJEMPP6NJ5KT7BQHDLTKKYDCK4NYPWA, seg_pos={0 0}, upload=188081]: Done uploading piece. err: <nil>
Mar 18 19:59:27.186328411 (195h48m30.201998019s) [noise=true , address=77.90.31.15:4700 , piece_id=PKZJFZCDED5EEU2JER2O4O572KTXWMU7VYS2E476VDM73T5N3JWQ, seg_pos={0 0}, upload=188081]: Done uploading piece. err: <nil>
Mar 18 19:59:27.186518549 (195h48m30.202188149s) [noise=true , address=45.91.116.7:4011 , piece_id=4Q5LRA5RL5RLR4XQ6RFZIBXJDJMWEK5SPUD5TBQIMZ337GH7PT6Q, seg_pos={0 0}, upload=188081]: Done uploading piece. err: <nil>
Mar 18 19:59:27.195717066 (195h48m30.211386672s) [noise=true , address=45.91.116.9:4263 , piece_id=NZWJQOKC6RQHYPWOFBKYYZ73ZF2C4KQXMEMT56F6ZR64L23SR4TQ, seg_pos={0 0}, upload=188081]: Done uploading piece. err: <nil>
Mar 18 19:59:27.208324140 (195h48m30.22399376s) [noise=true , address=77.90.31.13:4625 , piece_id=HYSJIJPCVIEN3UHIW4MS4SJONXQX4GLZBRAQD3M5LV7EFGFZEZJA, seg_pos={0 0}, upload=188081]: Done uploading piece. err: <nil>
Mar 18 19:59:27.267609377 (195h48m30.283278977s) [noise=true , address=79.127.219.38:4817 , piece_id=JE4QG7HWYDJGXZPUDW7BJSERCUBOUJGCQAKU6RLC3URPGOKCINAA, seg_pos={0 0}, upload=188081]: Done uploading piece. err: <nil>
Mar 18 19:59:52.988943339 (195h48m56.004612945s) [noise=true , address=145.79.227.47:4265 , piece_id=F7BSRBCLSISCSICRQYSN4CB7DTN6N77XXBGKLSURTCHUG4COOVUA, seg_pos={0 0}, upload=188081]: Done uploading piece. err: ecclient: upload failed (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ, address:145.79.227.47:4265): protocol: rpc: dial tcp 145.79.227.47:4265: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:52.989257522 (195h48m56.004927131s) [noise=true , address=145.79.227.10:4404 , piece_id=OM3WIFEMP37FY53GTCMAXS6ZEICTWCSVOBH26UFTPICIUAZNULFQ, seg_pos={0 0}, upload=188081]: Done uploading piece. err: ecclient: upload failed (node:12u1v9cAH2841Zq5JieXcPKWHYuEkg18S89ykFQuhtYBmvAPS7b, address:145.79.227.10:4404): protocol: rpc: dial tcp 145.79.227.10:4404: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:52.989342588 (195h48m56.005012197s) [noise=true , address=145.79.227.7:4298 , piece_id=N46DXUO65PDTHCVC4PZ37LBKN3JCTCTXCSIUT4ZNS3OBZNUEKXQQ, seg_pos={0 0}, upload=188081]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: rpc: dial tcp 145.79.227.7:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:52.989403415 (195h48m56.005073021s) [noise=true , address=145.79.227.33:4872 , piece_id=TBCLYWZ7PPKQTRMPFRIWQKE7NLHBDWPBDS2EFQ3KEBZQDW45HNXA, seg_pos={0 0}, upload=188081]: Done uploading piece. err: ecclient: upload failed (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec, address:145.79.227.33:4872): protocol: rpc: dial tcp 145.79.227.33:4872: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:52.989842410 (195h48m56.00551202s) [noise=true , address=145.79.227.48:4303 , piece_id=CBSK3ULMM5OB5W5UYEXNOJBMGL3AN5NFKUTOUV4OGO74MQGY7Y2Q, seg_pos={0 0}, upload=188081]: Done uploading piece. err: ecclient: upload failed (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT, address:145.79.227.48:4303): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:59:52.990049663 (195h48m56.005719269s) [noise=true , address=145.79.227.16:4913 , piece_id=6VA3TDQRU2AFSPF54A2QTYB633V5UATUH4EVGCDKDVRMYB7ZXEJA, seg_pos={0 0}, upload=188081]: Done uploading piece. err: ecclient: upload failed (node:15rBbsiT2s5kCYxgPgkai7dBHjfUu6mr4CXMWbYz7pFuJQCLnm, address:145.79.227.16:4913): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:59:52.990426911 (195h48m56.006096524s) [noise=true , address=145.79.227.13:4807 , piece_id=MYR24VBLJPY223FQKXGEBDUCF7NC47VLBIWCA5SMB4JCHKWMDELQ, seg_pos={0 0}, upload=188081]: Done uploading piece. err: ecclient: upload failed (node:1MZ9LJLHF4XXvmRG2r7XkanDfdNSKCUgua9EBPusPuGyLzTLo8, address:145.79.227.13:4807): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:59:52.990535119 (195h48m56.006204729s) [noise=true , address=145.79.227.14:4842 , piece_id=NV3IH3AVYGUROSSTZ54HKNS3LSACCR3TJVM2FDQOFOLTX6MQTWYQ, seg_pos={0 0}, upload=188081]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:145.79.227.14:4842): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:59:52.990576008 (195h48m56.006245595s) [seg_pos={0 0}, upload=188081]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Mar 18 19:59:53.109776118 (195h48m56.12544572s) [upload=188081]: Done uploading segments...
Mar 18 19:59:53.109823676 (195h48m56.125493281s) [upload=188081]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.124162443s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
30 out of 60 is failed (threshold=6) [.........................XXXXXXXX....XX.XXXXXXXXXXXXXXXXXXXX] duration:
5.001327754s trace:
https://jaeger.dev.storj.io/trace/612017d03f40e054
FAIL (2026-03-18 19:59:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00134543s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [.............................X..X...X..X..X........X..X..X..] duration:
20.653242708s trace:
https://jaeger.dev.storj.io/trace/657fdcaa051c01a8
FAIL (2026-03-18 19:59:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 13)
======LOGS======
18 19:56:59.292408464 (195h46m2.308078058s) [upload=188049]: Uploading segments...
Mar 18 19:56:59.292755979 (195h46m2.30842558s) [upload=188049]: Got next segment. Inline: false
Mar 18 19:56:59.292808462 (195h46m2.308478061s) [upload=188049]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:56:59.292813871 (195h46m2.308483471s) [upload=188049]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:57:02.876720251 (195h46m5.892389863s) [seg_pos={0 0}, upload=188049]: Begin upload segment...
Mar 18 19:57:02.876793971 (195h46m5.892463581s) [seg_pos={0 0}, upload=188049]: Done begin upload segment.
Mar 18 19:57:02.876833263 (195h46m5.892502873s) [noise=true , address=45.91.116.5:4313 , piece_id=NBWJEW5YG75SETFUT5GDB7SY7AV2FKSXDVYU6GDLH3LDBJWZIEXQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.876890974 (195h46m5.892560585s) [noise=true , address=45.91.116.8:4061 , piece_id=RVNNCRXRDGITJ4ANLAZWW5BIQQV6XZPA3664Z6YK4VYZ6WWFUGZQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.877956175 (195h46m5.893625785s) [noise=true , address=45.91.116.12:4252 , piece_id=3S6PWGR6TJLKHO4QNMWD2QC5POZFB7LDIMO2IWLQ6WMZQHYPGUGA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.878298073 (195h46m5.89396768s) [noise=true , address=167.17.72.70:4755 , piece_id=WXEJWO6LCSL4XPKNR3K2UQDD4REU6VDGXTSONMIXC2W5OHZD5FLA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.879712704 (195h46m5.895382315s) [noise=true , address=5.231.73.4:4023 , piece_id=ZG6PQBPOBCODENWWP2AQ3VWKT5JJ6DYYXET7EV4WOA6XLSNDUDJQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.880345930 (195h46m5.896015539s) [noise=true , address=79.127.205.235:4261 , piece_id=MMBAEATMIDZI3PIPWTANIPEFY6LWLD36WSRS4DGVP7GNO2THGDJA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.880748175 (195h46m5.896417775s) [noise=true , address=193.149.164.71:4315 , piece_id=AFPIVEIWVLXPXAVIAMBYEUCTE2IV4UMV72UCB632TB4WYS5PK56A, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.881290748 (195h46m5.896960358s) [noise=true , address=193.149.164.66:4991 , piece_id=JFQCEGUEYXYJ7RWJ7W2U73HGVNO2LFRLZ4YUKKSGXYQGGCCOHHYA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.882022155 (195h46m5.897691756s) [noise=true , address=77.90.31.19:4875 , piece_id=OPWPGCXK3DPV7WHTCY57VN5NWM3HXWSSZBOKNSYVY2LEHCRU665A, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.882473710 (195h46m5.898143318s) [noise=true , address=45.91.116.6:4006 , piece_id=DK5KUCW3YZIV5ATUEXO3C4O7GEPSLMTPVGZFZAGXIUF3TSBGW4TQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.883003620 (195h46m5.898673224s) [noise=true , address=145.79.227.28:4749 , piece_id=RAUWPP47L2SQHEPUVNTRZIWFPG7BNDD5DDYSUGI63K7IGI6QS7AA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.883504542 (195h46m5.899174149s) [noise=true , address=167.17.72.69:4738 , piece_id=Y5UBOMFHBSJEX7QLHTR3BSP5OVS7J4HQNUP7TPKPJHSU43JKWTWQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.884006474 (195h46m5.899676078s) [noise=true , address=145.79.227.18:4039 , piece_id=DWKTIPR4SKDEH43DTFIPQZGRVPMYNXGAH4FMAY2USNNTUU2WIHPA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.884343043 (195h46m5.900012643s) [noise=true , address=145.79.227.36:4538 , piece_id=XK7DMW633XPQX7RUL4JKHD75BDQOVVBYJ5SBCAU6X4GWWF4DZ5BA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.884660141 (195h46m5.900329745s) [noise=true , address=145.79.227.49:4343 , piece_id=EUYXULIELHYJGX6UKMMOIWTOUBS6YCGEPIXSMINDVCUBBRUBL34A, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.885052951 (195h46m5.900722557s) [noise=true , address=145.79.227.32:4846 , piece_id=PXUAYXKNTDYZUBQLWUPG5D6K2L4LT7N73BXRXEDARWC36NICFJOQ, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.886175686 (195h46m5.901845308s) [noise=true , address=193.149.164.69:4073 , piece_id=2WMFLFXQ6TMOLKFU3NXT5CCLE5GN5IZR2JQO2ZXA7MCMZWQOWLKA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:02.886594862 (195h46m5.902264463s) [noise=true , address=193.149.164.70:4133 , piece_id=G54QWDQDBWQKGBQFLWK6NT26CJIQFWEC4MCWSWTKYE3FMHR6FMRA, seg_pos={0 0}, upload=188049]: Uploading piece...
Mar 18 19:57:03.444101668 (195h46m6.459771278s) [upload=188049]: Next returned nil segment
Mar 18 19:57:03.444124472 (195h46m6.459794082s) [upload=188049]: Waiting for error group managing segments...
Mar 18 19:57:03.590092875 (195h46m6.605762476s) [noise=true , address=45.91.116.12:4252 , piece_id=3S6PWGR6TJLKHO4QNMWD2QC5POZFB7LDIMO2IWLQ6WMZQHYPGUGA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.603331674 (195h46m6.619001274s) [noise=true , address=45.91.116.6:4006 , piece_id=DK5KUCW3YZIV5ATUEXO3C4O7GEPSLMTPVGZFZAGXIUF3TSBGW4TQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.613170926 (195h46m6.628840529s) [noise=true , address=45.91.116.5:4313 , piece_id=NBWJEW5YG75SETFUT5GDB7SY7AV2FKSXDVYU6GDLH3LDBJWZIEXQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.617370351 (195h46m6.633039952s) [noise=true , address=45.91.116.8:4061 , piece_id=RVNNCRXRDGITJ4ANLAZWW5BIQQV6XZPA3664Z6YK4VYZ6WWFUGZQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.687019561 (195h46m6.702689162s) [noise=true , address=77.90.31.19:4875 , piece_id=OPWPGCXK3DPV7WHTCY57VN5NWM3HXWSSZBOKNSYVY2LEHCRU665A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.967066275 (195h46m6.982735877s) [noise=true , address=167.17.72.70:4755 , piece_id=WXEJWO6LCSL4XPKNR3K2UQDD4REU6VDGXTSONMIXC2W5OHZD5FLA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.977678776 (195h46m6.993348378s) [noise=true , address=5.231.73.4:4023 , piece_id=ZG6PQBPOBCODENWWP2AQ3VWKT5JJ6DYYXET7EV4WOA6XLSNDUDJQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:03.986006395 (195h46m7.001676s) [noise=true , address=167.17.72.69:4738 , piece_id=Y5UBOMFHBSJEX7QLHTR3BSP5OVS7J4HQNUP7TPKPJHSU43JKWTWQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.105385143 (195h46m7.121054743s) [noise=true , address=193.149.164.70:4133 , piece_id=G54QWDQDBWQKGBQFLWK6NT26CJIQFWEC4MCWSWTKYE3FMHR6FMRA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.113300230 (195h46m7.12896983s) [noise=true , address=193.149.164.71:4315 , piece_id=AFPIVEIWVLXPXAVIAMBYEUCTE2IV4UMV72UCB632TB4WYS5PK56A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.156743026 (195h46m7.172412637s) [noise=true , address=193.149.164.66:4991 , piece_id=JFQCEGUEYXYJ7RWJ7W2U73HGVNO2LFRLZ4YUKKSGXYQGGCCOHHYA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.212855060 (195h46m7.22852466s) [noise=true , address=79.127.205.235:4261 , piece_id=MMBAEATMIDZI3PIPWTANIPEFY6LWLD36WSRS4DGVP7GNO2THGDJA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:04.223915835 (195h46m7.239585447s) [noise=true , address=193.149.164.69:4073 , piece_id=2WMFLFXQ6TMOLKFU3NXT5CCLE5GN5IZR2JQO2ZXA7MCMZWQOWLKA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: <nil>
Mar 18 19:57:59.001554644 (195h47m2.017224275s) [noise=true , address=145.79.227.49:4343 , piece_id=EUYXULIELHYJGX6UKMMOIWTOUBS6YCGEPIXSMINDVCUBBRUBL34A, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:16EsEMRW53sjKVag2WpSDGLhELyE2bSXyW4aAurCvDFKRNkXuf, address:145.79.227.49:4343): protocol: rpc: dial tcp 145.79.227.49:4343: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.001715114 (195h47m2.017384752s) [noise=true , address=145.79.227.32:4846 , piece_id=PXUAYXKNTDYZUBQLWUPG5D6K2L4LT7N73BXRXEDARWC36NICFJOQ, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:1X6WKD6kuW1CgatniwYHDWrT9tfdq9sxB25UxAUBzTRXsJPyfH, address:145.79.227.32:4846): protocol: rpc: dial tcp 145.79.227.32:4846: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.002131998 (195h47m2.017801609s) [noise=true , address=145.79.227.18:4039 , piece_id=DWKTIPR4SKDEH43DTFIPQZGRVPMYNXGAH4FMAY2USNNTUU2WIHPA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:145.79.227.18:4039): protocol: rpc: dial tcp 145.79.227.18:4039: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.002164582 (195h47m2.017834216s) [noise=true , address=145.79.227.36:4538 , piece_id=XK7DMW633XPQX7RUL4JKHD75BDQOVVBYJ5SBCAU6X4GWWF4DZ5BA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:145.79.227.36:4538): protocol: rpc: dial tcp 145.79.227.36:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.002428976 (195h47m2.018098586s) [noise=true , address=145.79.227.28:4749 , piece_id=RAUWPP47L2SQHEPUVNTRZIWFPG7BNDD5DDYSUGI63K7IGI6QS7AA, seg_pos={0 0}, upload=188049]: Done uploading piece. err: ecclient: upload failed (node:1CKpmdJXXJVjy5PYy8N82gquvqGPaaLE3Qxz2pTAbTUPnpBZCi, address:145.79.227.28:4749): protocol: rpc: dial tcp 145.79.227.28:4749: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:57:59.002458729 (195h47m2.01812833s) [seg_pos={0 0}, upload=188049]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil>
Mar 18 19:57:59.118230032 (195h47m2.133899643s) [upload=188049]: Done uploading segments...
Mar 18 19:57:59.118301627 (195h47m2.133971217s) [upload=188049]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13)
Duration: 20.65326412s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXX] duration:
30.111588085s trace:
https://jaeger.dev.storj.io/trace/43a7e5940159e8af
FAIL (2026-03-18 19:59:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Mar 18 19:58:36.207916772 (195h47m39.223586376s) [upload=188072]: Starting upload
Mar 18 19:58:36.208014400 (195h47m39.223684004s) [upload=188072]: Done starting upload
Mar 18 19:58:36.208108476 (195h47m39.223778076s) [upload=188072]: Uploading segments...
Mar 18 19:58:36.208328211 (195h47m39.223997805s) [upload=188072]: Got next segment. Inline: false
Mar 18 19:58:36.208383755 (195h47m39.224053357s) [upload=188072]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Mar 18 19:58:36.208389469 (195h47m39.224059069s) [upload=188072]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Mar 18 19:58:38.944013895 (195h47m41.959683503s) [seg_pos={0 0}, upload=188072]: Begin upload segment...
Mar 18 19:58:38.944081102 (195h47m41.959750703s) [seg_pos={0 0}, upload=188072]: Done begin upload segment.
Mar 18 19:58:38.944114407 (195h47m41.959784013s) [noise=true , address=79.127.219.46:4891 , piece_id=YTLYMBY5HR2SLSNVJANWJQGP2YY7LQ6IGIRELPN3GZLANCY7WZSA, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.944126704 (195h47m41.959796314s) [noise=true , address=77.90.31.16:4738 , piece_id=3ZIDGBHXYKA52NYB23YSBMLMCDM3M7DU6F3DTUPVEPYOLWX6AXVA, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.944246420 (195h47m41.95991602s) [noise=true , address=79.127.219.38:4818 , piece_id=PPDGUJJCP4T6BZ2VTEMTPU37PL7W5GBKJ3DRUEE63O332XQMRARA, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.944628427 (195h47m41.960298037s) [noise=true , address=45.91.116.6:4976 , piece_id=ESXSGZEJTWTNLJANG6VS4HWMVFROXEW26VRELVADDTDVMKBM742A, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.945084981 (195h47m41.960754589s) [noise=true , address=77.90.31.10:4521 , piece_id=6XBRMQVXHO3NWBRZY4P3JBWAVUMJP2YYOL3FEKTMVGEVYNXOXYJQ, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.945476227 (195h47m41.96114583s) [noise=true , address=77.90.31.17:4774 , piece_id=V6OR6B6SKLDZNE2NVIMLNDFPI37RGSH4V4SZY2VTS4QOHLOQHUZQ, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.946063636 (195h47m41.96173324s) [noise=true , address=145.79.227.42:4122 , piece_id=EETOG2LHDVXSFMQDYDEGAXEJJ7LZQTW3P5IILXNB7R2YMODMDHVQ, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.946484600 (195h47m41.962154212s) [noise=true , address=145.79.227.37:4566 , piece_id=HCRUSBL764FDKVQX642HAVM42NF3RPKLPG262AQOZPNS2VNH5MKQ, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.946974438 (195h47m41.962644041s) [noise=true , address=45.91.116.12:4225 , piece_id=PS5DCN6XM75JHBTMPVFVZIPZOD4KLV4PUM3VOJK44KN2N2EWVAQA, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.947483629 (195h47m41.96315324s) [noise=true , address=145.79.227.23:4551 , piece_id=REDB3DVCSIZQ2SHJA7N73N4N6EE7PCDSCEX7AAABHW2NE3H2IISQ, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.948041959 (195h47m41.963711566s) [noise=true , address=145.79.227.3:4155 , piece_id=D5OVHAXTLTSPISYGGBW76SSAM4U7TW67XPTX2UOHWYVQSB6GLTHA, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.948548031 (195h47m41.964217631s) [noise=true , address=45.91.116.14:4196 , piece_id=T4RSDC3T7CAVOLZ5BEI6LTC26G5FBHPXUG2PMHM4IYX4DMUSG6VA, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.949971600 (195h47m41.965641209s) [noise=true , address=145.79.227.8:4334 , piece_id=OXU73OGZVYZ7ATZQXP5GLKRUVS7RNKN3OVKQWB4PG6S4OSHVL6EQ, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.951341438 (195h47m41.967011048s) [noise=true , address=45.91.116.10:4084 , piece_id=FQQ57METILCYVITRFJHYFIK4E2OLK3R2CYNL2YT2AD4FH4O7JXPQ, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.952826548 (195h47m41.968496153s) [noise=true , address=45.91.116.11:4119 , piece_id=7AG7TT75O7G4BUBY646XOBRDVGAA4HYVLOV547DFYSLE7GUEHSLA, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:38.954498378 (195h47m41.970167987s) [noise=true , address=45.91.116.5:4298 , piece_id=ZDMLOQG4PIBUX7KWNIAWB7TM5TPP7BGKNALLBSLSMTLKEU5JZXBQ, seg_pos={0 0}, upload=188072]: Uploading piece...
Mar 18 19:58:39.549975674 (195h47m42.565645294s) [upload=188072]: Next returned nil segment
Mar 18 19:58:39.549995626 (195h47m42.565665228s) [upload=188072]: Waiting for error group managing segments...
Mar 18 19:58:39.681990876 (195h47m42.697660479s) [noise=true , address=77.90.31.10:4521 , piece_id=6XBRMQVXHO3NWBRZY4P3JBWAVUMJP2YYOL3FEKTMVGEVYNXOXYJQ, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.691204305 (195h47m42.706873909s) [noise=true , address=45.91.116.12:4225 , piece_id=PS5DCN6XM75JHBTMPVFVZIPZOD4KLV4PUM3VOJK44KN2N2EWVAQA, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.692564050 (195h47m42.708233656s) [noise=true , address=45.91.116.5:4298 , piece_id=ZDMLOQG4PIBUX7KWNIAWB7TM5TPP7BGKNALLBSLSMTLKEU5JZXBQ, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.692737194 (195h47m42.708406794s) [noise=true , address=79.127.219.38:4818 , piece_id=PPDGUJJCP4T6BZ2VTEMTPU37PL7W5GBKJ3DRUEE63O332XQMRARA, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.696978658 (195h47m42.712648259s) [noise=true , address=45.91.116.11:4119 , piece_id=7AG7TT75O7G4BUBY646XOBRDVGAA4HYVLOV547DFYSLE7GUEHSLA, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.701890424 (195h47m42.717560027s) [noise=true , address=45.91.116.14:4196 , piece_id=T4RSDC3T7CAVOLZ5BEI6LTC26G5FBHPXUG2PMHM4IYX4DMUSG6VA, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.708182563 (195h47m42.723852165s) [noise=true , address=45.91.116.6:4976 , piece_id=ESXSGZEJTWTNLJANG6VS4HWMVFROXEW26VRELVADDTDVMKBM742A, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.715698147 (195h47m42.731367769s) [noise=true , address=45.91.116.10:4084 , piece_id=FQQ57METILCYVITRFJHYFIK4E2OLK3R2CYNL2YT2AD4FH4O7JXPQ, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.746216725 (195h47m42.761886326s) [noise=true , address=79.127.219.46:4891 , piece_id=YTLYMBY5HR2SLSNVJANWJQGP2YY7LQ6IGIRELPN3GZLANCY7WZSA, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.767185721 (195h47m42.782855325s) [noise=true , address=77.90.31.16:4738 , piece_id=3ZIDGBHXYKA52NYB23YSBMLMCDM3M7DU6F3DTUPVEPYOLWX6AXVA, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:58:39.781500452 (195h47m42.797170053s) [noise=true , address=77.90.31.17:4774 , piece_id=V6OR6B6SKLDZNE2NVIMLNDFPI37RGSH4V4SZY2VTS4QOHLOQHUZQ, seg_pos={0 0}, upload=188072]: Done uploading piece. err: <nil>
Mar 18 19:59:05.906858171 (195h48m8.9225278s) [noise=true , address=145.79.227.3:4155 , piece_id=D5OVHAXTLTSPISYGGBW76SSAM4U7TW67XPTX2UOHWYVQSB6GLTHA, seg_pos={0 0}, upload=188072]: Done uploading piece. err: ecclient: upload failed (node:1yiUBiCEpAs5yDmgVfFGuxgW4R53jzMJugo8Piff3p79EEZdWr, address:145.79.227.3:4155): protocol: rpc: dial tcp 145.79.227.3:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:05.907261101 (195h48m8.92293073s) [noise=true , address=145.79.227.23:4551 , piece_id=REDB3DVCSIZQ2SHJA7N73N4N6EE7PCDSCEX7AAABHW2NE3H2IISQ, seg_pos={0 0}, upload=188072]: Done uploading piece. err: ecclient: upload failed (node:12vR5WRMGoYyzYDZzwELFw6kLG6vtKsUeFM9kUhgZDLGKB6vcXs, address:145.79.227.23:4551): protocol: rpc: dial tcp 145.79.227.23:4551: i/o timeout; context deadline exceeded
Mar 18 19:59:05.907339687 (195h48m8.923009321s) [noise=true , address=145.79.227.37:4566 , piece_id=HCRUSBL764FDKVQX642HAVM42NF3RPKLPG262AQOZPNS2VNH5MKQ, seg_pos={0 0}, upload=188072]: Done uploading piece. err: ecclient: upload failed (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F, address:145.79.227.37:4566): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:59:05.907460470 (195h48m8.923130079s) [noise=true , address=145.79.227.42:4122 , piece_id=EETOG2LHDVXSFMQDYDEGAXEJJ7LZQTW3P5IILXNB7R2YMODMDHVQ, seg_pos={0 0}, upload=188072]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: context deadline exceeded; context deadline exceeded
Mar 18 19:59:05.907643160 (195h48m8.923312763s) [noise=true , address=145.79.227.8:4334 , piece_id=OXU73OGZVYZ7ATZQXP5GLKRUVS7RNKN3OVKQWB4PG6S4OSHVL6EQ, seg_pos={0 0}, upload=188072]: Done uploading piece. err: ecclient: upload failed (node:12KYMZAFsJbhWC4CdKRod2Z6qsC6ANcpqmpWSWx7UYajp6JvKH1, address:145.79.227.8:4334): protocol: rpc: dial tcp 145.79.227.8:4334: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Mar 18 19:59:05.907672002 (195h48m8.923341602s) [seg_pos={0 0}, upload=188072]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Mar 18 19:59:06.016221057 (195h48m9.031890658s) [upload=188072]: Done uploading segments...
Mar 18 19:59:06.016258091 (195h48m9.031927692s) [upload=188072]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.111616605s
core
uplink
us-select-1
|