Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...................................X..XX..XX................] duration:
1.623897249s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:45:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.623923325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X..X..................] duration:
2.237750421s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:45:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.237774256s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................X...XX...X................] duration:
2.70640967s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:45:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.706444616s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X..X.................] duration:
2.363686526s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:45:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.363729413s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X..X..................] duration:
2.246813504s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:45:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.24682819s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X..X................] duration:
2.372533944s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:45:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.372555531s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................................X...XX...X...............] duration:
3.13796648s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:45:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.138013686s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [....................................X..XX..XX...............] duration:
1.6362803s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:44:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.636302955s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.....................................X..XX..XX..............] duration:
1.614845314s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:44:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.614896375s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X..X.................] duration:
2.402338375s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:44:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.40236421s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X..X...............] duration:
2.252659919s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:44:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.252794989s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................................X...XX...X..............] duration:
2.78657603s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:44:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.786618359s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [......................................X..XX..XX.............] duration:
1.68673729s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:43:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.686757745s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X..X................] duration:
2.194076868s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:43:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.194105339s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................................X...XX...X.............] duration:
2.630134268s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:43:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.630159749s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X..X..............] duration:
2.318236467s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:43:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.318356078s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................................X...XX...X............] duration:
2.487072088s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:42:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.48757276s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.......................................X..XX..XX............] duration:
1.70987247s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:42:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.709908521s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X..X...............] duration:
2.33595065s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:42:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.335974849s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X..X.............] duration:
2.319607712s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:41:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.319637736s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [........................................X..XX..XX...........] duration:
1.703378792s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:41:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.703408225s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................................X...XX...X...........] duration:
2.486453968s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:41:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.486479856s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.........................................X..XX..XX..........] duration:
1.59653292s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:41:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.596853856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X..X..............] duration:
2.392099259s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:41:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.39215714s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................................X...XX...X..........] duration:
2.787311379s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:41:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.787410335s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X..X............] duration:
2.324263588s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:41:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.324297669s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..........................................X..XX..XX.........] duration:
1.575486389s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:40:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.575616914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X..X...........] duration:
2.311178578s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:40:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.311316084s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...........................................X..XX..XX........] duration:
1.636400579s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:40:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.636425777s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X..X.............] duration:
2.272750746s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:40:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.27277711s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................................X...XX...X.........] duration:
2.775633092s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:40:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.775666027s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X..X............] duration:
2.215008266s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:39:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.215034099s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X..X...........] duration:
2.33973742s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:39:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.339762425s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X..X..........] duration:
2.248339045s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:39:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.248378328s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [............................................X..XX..XX.......] duration:
1.545771519s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:39:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.545793364s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................................X...XX...X........] duration:
2.432586415s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:39:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.432678673s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X..X.........] duration:
2.441341623s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:39:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.441393318s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X..X........] duration:
2.336469212s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:38:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.336559295s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X..X..........] duration:
2.394630794s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:38:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.394683798s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................................X...XX...X.......] duration:
2.68229664s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:38:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.682323992s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.............................................X..XX..XX......] duration:
1.569301695s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:38:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.569320923s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X..X.......] duration:
2.292434897s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:38:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.292459376s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X..X......] duration:
2.321311631s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:38:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.321330054s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X..X.........] duration:
2.44962622s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:38:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.44965086s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X..X.....] duration:
2.307358458s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:37:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.307402018s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................................X...XX...X......] duration:
2.619362561s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:37:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.619395619s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................................X...XX...X.....] duration:
2.56096201s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:37:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.560979016s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X..X........] duration:
2.395895134s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:37:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.395923211s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..............................................X..XX..XX.....] duration:
1.677846466s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:37:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.677872795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X..X....] duration:
2.286191076s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:37:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.286232857s
core
uplink
us-select-1
us1-placement-20
|