Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X..X..........................................] duration:
2.299830179s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:04:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.299866448s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X..X............................................] duration:
2.227297541s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:04:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.227327028s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X...XX...X..........................................] duration:
2.78733185s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:04:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.787471134s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...........X..XX..XX........................................] duration:
1.656469746s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:03:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.656491019s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X..X.........................................] duration:
2.441050016s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:03:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.441085622s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X...XX...X.........................................] duration:
2.743217838s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:03:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.74324787s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X..X...........................................] duration:
2.221425404s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:02:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.221446969s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [............X..XX..XX.......................................] duration:
1.556555505s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:02:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.556579786s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........X...XX...X........................................] duration:
2.614869795s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:02:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.614891941s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X..X........................................] duration:
2.53287528s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:02:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.532901808s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.............X..XX..XX......................................] duration:
1.576822347s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:02:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.576848962s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X..X..........................................] duration:
2.224985382s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:02:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.225099144s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..............X..XX..XX.....................................] duration:
1.567062315s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:01:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.567084209s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........X...XX...X.......................................] duration:
2.761118186s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:01:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.761147218s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X..X.........................................] duration:
2.242650561s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:01:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.242683657s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X..X.......................................] duration:
2.216393339s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:01:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.216433383s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...............X..XX..XX....................................] duration:
2.114286892s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:01:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.114308728s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X..X........................................] duration:
2.197409281s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:01:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.197505517s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............X...XX...X......................................] duration:
2.695047681s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:00:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.69508974s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................X..X.......................................] duration:
2.281541223s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:00:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.281567875s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X..X......................................] duration:
2.336447138s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 13:00:41 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.33647482s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [................X..XX..XX...................................] duration:
1.593530182s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 13:00:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.593560142s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............X...XX...X.....................................] duration:
2.529415562s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 13:00:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.529441183s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................X..X......................................] duration:
2.248405325s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 13:00:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.248434657s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.................X..XX..XX..................................] duration:
1.593693407s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:59:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.593717251s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X..X.....................................] duration:
2.330016425s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:59:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.330043313s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................X..X.....................................] duration:
2.257018534s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:59:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.257043325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............X...XX...X....................................] duration:
2.857042492s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:59:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.85707319s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X..X....................................] duration:
2.270924369s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:58:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.270949195s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............X...XX...X...................................] duration:
2.678187404s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:58:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.67821288s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..................X..XX..XX.................................] duration:
1.603561947s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:58:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.603591378s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................X..X....................................] duration:
2.267934528s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:58:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.267962856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X..X...................................] duration:
2.382635759s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:57:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.382662391s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................X..X...................................] duration:
2.242340431s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:57:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.242364755s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...................X..XX..XX................................] duration:
1.573100991s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:57:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.573145952s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................X...XX...X..................................] duration:
2.624794032s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:57:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.624813856s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X..X..................................] duration:
2.168196526s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:57:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.168220103s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [....................X..XX..XX...............................] duration:
1.615389186s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:57:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.615414507s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................X...XX...X.................................] duration:
2.721980552s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:57:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.722017626s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................X..X..................................] duration:
2.369689247s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:57:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.369716749s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................X...XX...X................................] duration:
2.644390562s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:56:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.644432228s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X..X.................................] duration:
2.414063004s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:56:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.414092407s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.....................X..XX..XX..............................] duration:
1.60735632s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:56:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.607382399s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................X..X.................................] duration:
2.216013688s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:56:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.216040163s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [......................X..XX..XX.............................] duration:
2.1502121s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:56:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.150237964s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................X...XX...X...............................] duration:
2.924397839s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:56:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.924421912s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X..X................................] duration:
2.409995644s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:55:53 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.410023483s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X..X................................] duration:
2.348198938s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:55:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.348223523s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................X...XX...X..............................] duration:
2.513446209s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:55:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.51347422s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X..X...............................] duration:
2.252223585s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:55:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.252342749s
core
uplink
us-select-1
us1-placement-20
|