Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.......................X..XX..XX............................] duration:
1.588688916s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:54:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.588711798s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X..X...............................] duration:
2.31893253s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:54:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.318960046s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................X...XX...X.............................] duration:
2.752142942s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:54:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.75217079s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X..X..............................] duration:
2.271050246s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:54:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.271076071s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................X...XX...X............................] duration:
2.625624085s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:54:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.62570198s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [........................X..XX..XX...........................] duration:
1.558686361s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:53:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.558715121s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X..X..............................] duration:
2.249913231s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:53:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.249942374s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X..X.............................] duration:
2.348792902s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:53:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.348823172s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................X...XX...X...........................] duration:
2.932395831s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:53:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.933111536s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.........................X..XX..XX..........................] duration:
1.659324437s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:53:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.659377998s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................X...XX...X..........................] duration:
2.752187788s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:53:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.752212002s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X..X.............................] duration:
2.369357771s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:53:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.369452438s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..........................X..XX..XX.........................] duration:
1.629318824s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:52:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.629356036s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...........................X..XX..XX........................] duration:
1.629025421s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:52:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.629049276s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X..X............................] duration:
2.212291185s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:52:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.212316402s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X..X............................] duration:
2.424432602s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:52:34 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.424459248s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [............................X..XX..XX.......................] duration:
1.656305725s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:52:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.656330047s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X..X...........................] duration:
2.286198411s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:52:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.28622311s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X..X...........................] duration:
2.205972172s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:51:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.206001231s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................X...XX...X.........................] duration:
2.625423245s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:51:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.626452489s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X..X..........................] duration:
2.32349354s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:51:46 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.323597579s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.............................X..XX..XX......................] duration:
2.354514894s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:51:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.354539914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X..X..........................] duration:
2.344244477s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:51:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.344268342s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X..X.........................] duration:
2.313322707s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:51:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.313423309s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................X...XX...X........................] duration:
2.644794926s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:50:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.64481068s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..............................X..XX..XX.....................] duration:
1.672606809s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:50:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.672660134s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................X..X.........................] duration:
2.245440806s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:50:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.245464665s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X..X........................] duration:
2.230983261s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:50:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.231011538s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................X...XX...X.......................] duration:
2.759383302s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:50:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.759421272s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X..X.......................] duration:
2.298601654s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:49:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.298629274s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................X...XX...X......................] duration:
2.782395195s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:49:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.782422269s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...............................X..XX..XX....................] duration:
1.627957313s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:49:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.627984247s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................X..X........................] duration:
2.365216701s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:49:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.365242515s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X..X......................] duration:
2.226132851s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:49:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.226152053s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................X...XX...X.....................] duration:
2.752297545s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:49:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.752359923s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X..X.....................] duration:
2.28528503s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:48:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.285313568s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [................................X..XX..XX...................] duration:
2.087737341s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:48:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.087802918s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................X..X.......................] duration:
2.343066322s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:48:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.34309477s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................X...XX...X....................] duration:
2.741932055s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:48:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.741951621s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X..X......................] duration:
2.201287118s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:48:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.201328364s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X..X.....................] duration:
2.348617367s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:47:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.348663419s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.................................X..XX..XX..................] duration:
1.58679128s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:47:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.586873789s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X..X....................] duration:
2.235279463s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:47:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.235306817s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................X...XX...X...................] duration:
2.732234033s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:47:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.732257305s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..................................X..XX..XX.................] duration:
1.559626292s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:46:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.559650915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................X..X....................] duration:
2.268320831s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:46:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.268335931s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X..X...................] duration:
2.207508071s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:46:54 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.207536911s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................X...XX...X..................] duration:
2.598175419s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:46:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.598210962s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X..X...................] duration:
2.285714462s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:46:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.285743658s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................X...XX...X.................] duration:
2.774961099s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:46:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.774976233s
edge
gateway
us-select-1
|