Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................................................X.X.] duration:
1.682187853s trace:
https://jaeger.dev.storj.io/trace/34563bca7399efcd
FAIL (2026-02-18 19:13:14 UTC)
orbiter warning: context deadline exceeded
Duration: 1.682214446s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................................................X.X] duration:
5.000669348s trace:
https://jaeger.dev.storj.io/trace/34563bca7399efcd
FAIL (2026-02-18 19:12:17 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00069623s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
3.566150376s trace:
https://jaeger.dev.storj.io/trace/15a4f3b68de38300
FAIL (2026-02-18 19:12:00 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 3.566176097s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.000410077s trace:
https://jaeger.dev.storj.io/trace/15a4f3b68de38300
FAIL (2026-02-18 19:11:45 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded
Duration: 5.000429363s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
3.310317887s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:15:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.310347205s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
3.07964356s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:15:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.079668849s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [XXX.........................................................] duration:
3.039404363s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:14:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.039432685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [XXXX........................................................] duration:
4.189788916s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:13:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 4.18985891s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.XXXX.......................................................] duration:
3.328994312s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:12:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.329022162s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [X.XXXX......................................................] duration:
3.079737283s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:12:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.07977031s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [XX.XXXX.....................................................] duration:
3.106410056s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:11:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.106439199s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [XXX.XXXX....................................................] duration:
3.485072202s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:10:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.485106364s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
8 out of 60 is failed (threshold=3) [XXXX.XXXX...................................................] duration:
2.913643146s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:10:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 2.913748339s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
9 out of 60 is failed (threshold=3) [XXXXX.XXXX..................................................] duration:
3.122572958s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:09:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.122602284s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
10 out of 60 is failed (threshold=3) [XXXXXX.XXXX.................................................] duration:
3.147955281s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:08:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.147986429s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
11 out of 60 is failed (threshold=3) [XXXXXXX.XXXX................................................] duration:
3.274537532s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:08:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.274567863s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
12 out of 60 is failed (threshold=3) [XXXXXXXX.XXXX...............................................] duration:
3.039876434s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:07:44 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.039974878s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
13 out of 60 is failed (threshold=3) [XXXXXXXXX.XXXX..............................................] duration:
3.394705462s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:06:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.394737676s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
14 out of 60 is failed (threshold=3) [XXXXXXXXXX.XXXX.............................................] duration:
3.30159931s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:05:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.301628364s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
15 out of 60 is failed (threshold=3) [XXXXXXXXXXX.XXXX............................................] duration:
3.342540373s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:04:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.342574721s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
16 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.XXXX...........................................] duration:
3.231181648s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:03:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.23127724s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
17 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.XXXX..........................................] duration:
3.493761049s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:03:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.493789154s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
18 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XXXX.........................................] duration:
3.062490831s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:02:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.062520829s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XXXX........................................] duration:
3.504250273s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:01:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.504299811s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
20 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XXXX.......................................] duration:
3.182109463s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:01:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.182146571s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XXXX......................................] duration:
3.109367171s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-18 00:00:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.109398068s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
21 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXX.XXXX.....................................] duration:
2.730344767s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:59:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 2.730372476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
22 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXX.XXXX....................................] duration:
3.032444133s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:59:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.032532246s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
22 out of 60 is failed (threshold=3) [.X.XXXXXXXXXXXXXXXXX.XXXX...................................] duration:
3.33737455s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:58:12 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.33740592s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
22 out of 60 is failed (threshold=3) [..X.XXXXXXXXXXXXXXXXX.XXXX..................................] duration:
2.929096029s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:57:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 2.929120947s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
22 out of 60 is failed (threshold=3) [...X.XXXXXXXXXXXXXXXXX.XXXX.................................] duration:
3.217651953s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:56:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.217680747s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
23 out of 60 is failed (threshold=3) [X...X.XXXXXXXXXXXXXXXXX.XXXX................................] duration:
3.211163314s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:56:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.21119275s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
24 out of 60 is failed (threshold=3) [XX...X.XXXXXXXXXXXXXXXXX.XXXX...............................] duration:
3.284279366s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:55:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.284380248s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
24 out of 60 is failed (threshold=3) [.XX...X.XXXXXXXXXXXXXXXXX.XXXX..............................] duration:
3.455973827s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:54:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.455998733s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
25 out of 60 is failed (threshold=3) [X.XX...X.XXXXXXXXXXXXXXXXX.XXXX.............................] duration:
3.334256451s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:54:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.334287864s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [XX.XX...X.XXXXXXXXXXXXXXXXX.XXXX............................] duration:
3.547034156s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:52:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.547060592s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [.XX.XX...X.XXXXXXXXXXXXXXXXX.XXXX...........................] duration:
3.387309091s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:52:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.387338735s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [..XX.XX...X.XXXXXXXXXXXXXXXXX.XXXX..........................] duration:
3.089750604s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:51:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.089774043s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [...XX.XX...X.XXXXXXXXXXXXXXXXX.XXXX.........................] duration:
3.209280627s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:50:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.209316906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [....XX.XX...X.XXXXXXXXXXXXXXXXX.XXXX........................] duration:
3.04625936s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:49:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.046287683s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [.....XX.XX...X.XXXXXXXXXXXXXXXXX.XXXX.......................] duration:
3.026809762s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:49:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.026904234s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [......XX.XX...X.XXXXXXXXXXXXXXXXX.XXXX......................] duration:
3.389781297s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:48:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.389807461s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
1.643659431s trace:
https://jaeger.dev.storj.io/trace/4f259542fd8bf4d2
FAIL (2026-02-17 23:47:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.64368878s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.999628932s trace:
https://jaeger.dev.storj.io/trace/478c1cba3de9cc23
FAIL (2026-02-17 23:47:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.999650975s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [.......XX.XX...X.XXXXXXXXXXXXXXXXX.XXXX.....................] duration:
3.024158981s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:47:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.024188344s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [XX..........................................................] duration:
1.661347999s trace:
https://jaeger.dev.storj.io/trace/4f259542fd8bf4d2
FAIL (2026-02-17 23:46:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.661362102s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [XXX.........................................................] duration:
1.907563849s trace:
https://jaeger.dev.storj.io/trace/4f259542fd8bf4d2
FAIL (2026-02-17 23:46:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.907592746s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
2.93862933s trace:
https://jaeger.dev.storj.io/trace/478c1cba3de9cc23
FAIL (2026-02-17 23:46:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.938890382s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
26 out of 60 is failed (threshold=3) [........XX.XX...X.XXXXXXXXXXXXXXXXX.XXXX....................] duration:
3.506733432s trace:
https://jaeger.dev.storj.io/trace/7f908743098f0576
FAIL (2026-02-17 23:46:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
7261153 (3h16m15.601715291s) [upload=2852]: Got next segment. Inline: false
Feb 17 23:33:10.667314810 (3h16m15.601768947s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Feb 17 23:33:10.667321427 (3h16m15.601775597s) [upload=2852]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Feb 17 23:33:10.776607940 (3h16m15.71106208s) [seg_pos={0 0}, upload=2852]: Begin upload segment...
Feb 17 23:33:10.776696742 (3h16m15.711150866s) [seg_pos={0 0}, upload=2852]: Done begin upload segment.
Feb 17 23:33:10.776760025 (3h16m15.711214166s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778171620 (3h16m15.712625756s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778280870 (3h16m15.712735011s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778678541 (3h16m15.71313268s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.778792620 (3h16m15.713246763s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779139233 (3h16m15.713593373s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779180385 (3h16m15.713634522s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779530340 (3h16m15.713984463s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779548787 (3h16m15.714002929s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779875552 (3h16m15.714329686s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.779903904 (3h16m15.71435804s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780300286 (3h16m15.714754424s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.780855656 (3h16m15.715309795s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781278384 (3h16m15.715732543s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781392731 (3h16m15.715846873s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.781698983 (3h16m15.716153121s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.782036364 (3h16m15.716490502s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:10.783236453 (3h16m15.717690579s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Uploading piece...
Feb 17 23:33:11.325135194 (3h16m16.25958933s) [upload=2852]: Next returned nil segment
Feb 17 23:33:11.325180688 (3h16m16.259634829s) [upload=2852]: Waiting for error group managing segments...
Feb 17 23:33:11.478370676 (3h16m16.412824815s) [noise=true , address=145.79.227.46:4617 , piece_id=FOI4GGL26BC3SLDV6EBYJQUDAGV2KTUPO6H2ONHNHII3ASD7DBWQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.489702119 (3h16m16.424156258s) [noise=true , address=145.79.227.2:4148 , piece_id=AZPLYCDJZG6YCSVF74C7OKLQ7X72ACSB2LR56YLS54TAVCARKOPQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.492159746 (3h16m16.426613883s) [noise=true , address=45.91.116.13:4161 , piece_id=JNFCHBJQWF3ZH7MZOXIGGTU3TI7YDBLZGB2OT5GEMIVYSN7QZVDA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.520249445 (3h16m16.454703572s) [noise=true , address=145.79.227.5:4256 , piece_id=36DCUTC75KOJFGCSLCSYD5FFDE34NQMCQ3K33G7C7DBKKKIWUESQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.533706957 (3h16m16.46816108s) [noise=true , address=145.79.227.19:4101 , piece_id=TG7I4JAMR2UAI5H2A7VBLRJBD5MYFKB5XXZBOFJLKFJAYE5AKV6Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:11.586898275 (3h16m16.5213524s) [noise=true , address=145.79.227.41:4097 , piece_id=5576YCUAGDCKXMKUDVQXC2UTKSRRRIXWHYVNG47HDVMHLAYXZTSQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011356848 (3h16m16.945810981s) [noise=true , address=193.149.164.70:4107 , piece_id=JHXBFJBLSI4HQ4JGCIBBPZ4Q4AA3O5NX5C5UIC3C4IAE2FWZ7GZQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.011462569 (3h16m16.945916731s) [noise=true , address=193.149.164.69:4093 , piece_id=735ICYWJIZOKH3HBZ26KU6C2SJC6IVEMIX2BMGOLOILIVAZQZLYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.020460262 (3h16m16.954914395s) [noise=true , address=193.149.164.66:4983 , piece_id=LO4RHURY4HLD3W6YMFO6B7P7MUXCTGJZSPMUOWHY3AXNJ7EFBYZA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.025949884 (3h16m16.960404006s) [noise=true , address=193.149.164.67:4018 , piece_id=5VUO47PF4A2JQX2U323VHGL5ZQR6W3O3M2ECWYRFWWVMLBZEDRHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.039088110 (3h16m16.973542232s) [noise=true , address=64.49.13.140:4510 , piece_id=CIGWVYBMKE3MXBDU3MJGQENEXPRM43B3DMOVLIUCT6GJJ323UFHQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:33:12.425365568 (3h16m17.359819691s) [noise=true , address=145.79.227.30:4768 , piece_id=USGQQFK6T555AA2BGQFN3M3CRY4S2MB7YKCP7VWG3MREL2DFTJ2Q, seg_pos={0 0}, upload=2852]: Done uploading piece. err: <nil>
Feb 17 23:34:10.399361497 (3h17m15.333815691s) [noise=true , address=77.90.31.16:4750 , piece_id=YT7RXY626TDCW33K4TFBAYQ6HR4QG6RHMCTU6SUPHJITOITPP6PA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399572798 (3h17m15.334026935s) [noise=true , address=77.90.31.13:4632 , piece_id=UPXZMVVQKHVEKRLOQOUI3NRFVPZ6EVV3GPPMZAINHATAFRTNOBUA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:13iPed9zYrthvh36xo3k4gsqm4tz86iVkNvAr3wauuArQLquyS, address:77.90.31.13:4632): protocol: rpc: dial tcp 77.90.31.13:4632: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399592421 (3h17m15.334046562s) [noise=true , address=77.90.31.14:4669 , piece_id=U5I622KDIZ7HDTQUJM7CMDS6DJ3LPDJKWYZKVVHDQTY3Z6URWXYA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T, address:77.90.31.14:4669): protocol: rpc: dial tcp 77.90.31.14:4669: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399635924 (3h17m15.334090042s) [noise=true , address=77.90.31.12:4624 , piece_id=5QYKED24FNXRXY3M5K5FLKOC6GN3GKYTOS3NCWP36QXPNYEAAENA, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: rpc: dial tcp 77.90.31.12:4624: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399794086 (3h17m15.334248222s) [noise=true , address=77.90.31.11:4581 , piece_id=J2Z6JEPECHBXOHQRLSORGZFLMFAHKXAXH2RHZV44B3EASHJRXZ7A, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp, address:77.90.31.11:4581): protocol: write tcp 139.59.144.22:54212->77.90.31.11:4581: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399835370 (3h17m15.334289512s) [noise=true , address=77.90.31.10:4532 , piece_id=MALUR6KD73HNUAS7RBXKFA2QMDIKA7NQBGFNVYMEX7G53LMY2WMQ, seg_pos={0 0}, upload=2852]: Done uploading piece. err: ecclient: upload failed (node:1T6kkKvTNR5MZZijjr1vMU6g75YkoRpzMvGSgQCqRnfCm9JULe, address:77.90.31.10:4532): protocol: rpc: dial tcp 77.90.31.10:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Feb 17 23:34:10.399875448 (3h17m15.334329567s) [seg_pos={0 0}, upload=2852]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Feb 17 23:34:10.497548320 (3h17m15.432002443s) [upload=2852]: Done uploading segments...
Feb 17 23:34:10.497582479 (3h17m15.432036614s) [upload=2852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 3.50678476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
2.778976379s trace:
https://jaeger.dev.storj.io/trace/478c1cba3de9cc23
FAIL (2026-02-17 23:45:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.77900904s
core
uplink
us-select-1
us1-placement-20
|