Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X...X.......................................................] duration:
2.725242301s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:13:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.725270582s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
2.252091851s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:13:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.252120941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..XX..XX....................................................] duration:
1.602573567s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:13:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.602598557s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
2.295437942s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:12:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.295459776s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [XX...X......................................................] duration:
2.647684272s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:12:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.647708737s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X..X....................................................] duration:
2.342938658s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:12:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.342972415s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.XX...X.....................................................] duration:
2.978972675s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:12:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.979017766s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
2.255378885s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:12:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.255406891s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration:
2.245697399s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:11:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.245735865s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [X..XX..XX...................................................] duration:
1.580293014s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:11:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.580319778s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X..X........................................................] duration:
2.166608347s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:11:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.166636816s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X..X.......................................................] duration:
2.279143874s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:11:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.279172361s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.X..XX..XX..................................................] duration:
1.69581739s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:11:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.695861185s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..XX...X....................................................] duration:
2.703850276s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:11:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.703872475s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X..X..................................................] duration:
2.278985706s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:10:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.279014354s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..X..XX..XX.................................................] duration:
1.618761983s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:10:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.618782394s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X..X......................................................] duration:
2.302474528s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:10:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.302507762s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...XX...X...................................................] duration:
2.773919155s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:10:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.773954529s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X...XX...X..................................................] duration:
2.846263768s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:10:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.846347794s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...X..XX..XX................................................] duration:
2.138778021s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:09:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.138807291s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...X..X.....................................................] duration:
2.327702106s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:09:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.327808322s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X..X.................................................] duration:
2.343373162s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:09:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.34340106s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X...XX...X.................................................] duration:
2.755866664s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:09:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.755894937s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [....X..XX..XX...............................................] duration:
1.642117958s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:09:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.642146045s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....X..X....................................................] duration:
2.217613644s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:08:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.217644785s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.....X..XX..XX..............................................] duration:
1.602136435s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:08:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.602160186s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X...XX...X................................................] duration:
2.454946974s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:08:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.454971199s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....X..X...................................................] duration:
2.271695157s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:08:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.271724878s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X..X................................................] duration:
2.301183802s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:08:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.301207814s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X...XX...X...............................................] duration:
2.904802064s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:08:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.904827264s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration:
2.330273171s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:08:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.330305458s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X..X..................................................] duration:
2.21696767s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:07:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.217043309s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [......X..XX..XX.............................................] duration:
2.099323981s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:07:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.099351971s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X..X.................................................] duration:
2.243285934s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:07:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.243315162s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.......X..XX..XX............................................] duration:
1.667319921s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:07:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.66736072s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X...XX...X..............................................] duration:
2.631995764s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:07:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.632022683s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X..X..............................................] duration:
2.37102331s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:07:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.371048253s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X..X................................................] duration:
2.272940863s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:06:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.272963628s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [........X..XX..XX...........................................] duration:
1.689370576s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:06:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.689394509s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X...XX...X.............................................] duration:
2.836964655s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:06:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.836990163s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X..X...............................................] duration:
2.384124535s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:06:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.384149346s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X...XX...X............................................] duration:
2.684497609s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:05:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.684522992s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X..X.............................................] duration:
2.241071352s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:05:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.241105545s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.........X..XX..XX..........................................] duration:
1.588218335s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:05:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.58824491s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X..X..............................................] duration:
2.250840042s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:05:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.250922877s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X..X............................................] duration:
2.345025647s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:05:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.34508384s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X..X.............................................] duration:
2.188307201s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:05:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.188334663s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X...XX...X...........................................] duration:
2.692466072s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:05:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.69249269s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X..X...........................................] duration:
2.286814158s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:04:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.28683723s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..........X..XX..XX.........................................] duration:
1.680372784s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:04:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.680397138s
core
uplink
us-select-1
us1-placement-20
|