Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................................X...XX...X......] duration:
2.619362561s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:37:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.619395619s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................................X...XX...X.....] duration:
2.56096201s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:37:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.560979016s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X..X........] duration:
2.395895134s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:37:13 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.395923211s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..............................................X..XX..XX.....] duration:
1.677846466s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:37:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.677872795s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X..X....] duration:
2.286191076s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:37:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.286232857s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...............................................X..XX..XX....] duration:
1.584147359s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:36:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.584172771s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X..X.......] duration:
2.376438555s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:36:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.376468018s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................................X...XX...X....] duration:
2.728186911s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:36:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.728209571s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X..X...] duration:
2.356021939s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:36:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.356054581s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [................................................X..XX..XX...] duration:
1.598277697s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:35:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.598299442s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X..X......] duration:
2.321397437s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:35:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.321426804s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................................X...XX...X...] duration:
2.648195275s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:35:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.648217874s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X..X.....] duration:
2.277549055s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:34:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.277586926s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.................................................X..XX..XX..] duration:
1.622358826s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:34:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.622377903s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X..X..] duration:
2.22458532s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:34:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.224611246s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................................X...XX...X..] duration:
2.61377241s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:34:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.613805838s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X..X....] duration:
2.450489444s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:34:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.450517141s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................................X...XX...X.] duration:
2.830133089s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:34:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.83016245s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration:
2.356043678s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:34:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.356064664s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................X..X] duration:
1m0.109711273s trace:
https://jaeger.dev.storj.io/trace/39e6d886131375da
FAIL (2025-11-22 12:33:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
[noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.570689076 (266h48m43.929347736s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571011471 (266h48m43.929670125s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571034592 (266h48m43.929693252s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571407426 (266h48m43.930066082s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571483461 (266h48m43.930142106s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571758948 (266h48m43.930417588s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.571811091 (266h48m43.930469735s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572138300 (266h48m43.930796959s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572143829 (266h48m43.930802485s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572540606 (266h48m43.931199248s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:58.572956921 (266h48m43.931615565s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Uploading piece...
Nov 22 12:32:59.131933549 (266h48m44.490592208s) [upload=235275]: Next returned nil segment
Nov 22 12:32:59.131949897 (266h48m44.490608554s) [upload=235275]: Waiting for error group managing segments...
Nov 22 12:32:59.271693335 (266h48m44.630351997s) [noise=true , address=145.79.227.52:4480 , piece_id=6KWF64FY7O3VE5IT3GZR5PUMC6MHYCFKEIJA6DSDYTR43NDDXNVQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.272782925 (266h48m44.631441579s) [noise=true , address=145.79.227.12:4771 , piece_id=HV43FRRZDWZIGF3RVWS4IMCP4G23MG66RGDXQGN3VXUYEZIIOFZQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.278008069 (266h48m44.636666723s) [noise=true , address=145.79.227.25:4629 , piece_id=BW476PXLCEDQOQ6SUWLKFUXRALHOEZCTWN4SSEHB72QSOHOHKS2A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.814658675 (266h48m45.173317315s) [noise=true , address=64.49.13.27:4967 , piece_id=DHXQO6LF2SKFAGCWE7MMWK4HS3A5ZP2OWVP3BCXZCTEA6MR3UTWQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:32:59.866926293 (266h48m45.225584934s) [noise=true , address=64.49.13.140:4506 , piece_id=QPVIKEWFNKTA4FSLOVETDYREPAWKF7HLZXETFEXPGQF6N4UXO62A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: <nil>
Nov 22 12:33:58.195052484 (266h49m43.553711146s) [noise=true , address=45.91.116.10:4098 , piece_id=3GURDQBSEZY7OID5GNVVU455GYJRNN24EHPK2ZMXKZNCK2UHWKYA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12Ji2HtjJ47F98PDMF35jMakxPqNMRBFGRQpNP8sWKrGdWxe2Yb, address:45.91.116.10:4098): protocol: rpc: dial tcp 45.91.116.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195264331 (266h49m43.553922993s) [noise=true , address=77.90.31.16:4762 , piece_id=PK54P4DDEZ4IUSSDSGU5C2CT62SXKEPJQMYTFQEZM2PGCNJ4UXKQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR, address:77.90.31.16:4762): protocol: rpc: dial tcp 77.90.31.16:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195513773 (266h49m43.554172435s) [noise=true , address=77.90.31.10:4540 , piece_id=6WA7K2P7XVLNYLPDGCUZLXTWIJACKCXKTVERCMIDZJE5H4IRMQ6A, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: rpc: dial tcp 77.90.31.10:4540: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.195825736 (266h49m43.554484399s) [noise=true , address=77.90.31.12:4610 , piece_id=DPXTV26RARVFEPPLAVD5YWKF2LG6SEGOJO2EC7TJWFSM7GB5M3BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, address:77.90.31.12:4610): protocol: rpc: dial tcp 77.90.31.12:4610: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196148729 (266h49m43.554807387s) [noise=true , address=77.90.31.21:4903 , piece_id=OZZY643LYTQ3TMWRBBRQKWLV54Y73KVRAEOWE4UDVVKHP5PHSYEQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1rZYWQ1a2sTKnFpF28iQVrem2s8nSpXkeJzZDaS7jvTZNANdfp, address:77.90.31.21:4903): protocol: write tcp 139.59.144.22:36336->77.90.31.21:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196443202 (266h49m43.555101858s) [noise=true , address=45.91.116.6:4986 , piece_id=6NS47VU53COX5GZLRVZ2SKE2OXAC3FUYOMMJW6GRYBQJMH7RVISQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12t1cuEr8MRLBbJ4GNSdYBxdhNQ6xDbTr8PHoEuVhMqh1ksgWon, address:45.91.116.6:4986): protocol: write tcp 139.59.144.22:32918->45.91.116.6:4986: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196513189 (266h49m43.555171849s) [noise=true , address=45.91.116.8:4077 , piece_id=HOZT3JWQO42BTCNWWDF7N543G5GR7PNOUYDITQL2YWPIXD6WI2DA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1ca5vTH8FegP7bkKHmXdh83iFE1s4BhSvLntWLqADUS2kadXEn, address:45.91.116.8:4077): protocol: rpc: dial tcp 45.91.116.8:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196739140 (266h49m43.555397797s) [noise=true , address=77.90.31.17:4769 , piece_id=IDYSTNXDZSGSOF2VPOGO574KWT2I2BJNSHILIT7SDM4L6O27Z2BQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: rpc: dial tcp 77.90.31.17:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.196795485 (266h49m43.555454141s) [noise=true , address=77.90.31.14:4686 , piece_id=ZIW4FNJU6TBZZG7VXJZRFKR7JPQCBRRO2E3AAFVVARFD3TN3YALA, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, address:77.90.31.14:4686): protocol: rpc: dial tcp 77.90.31.14:4686: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197340611 (266h49m43.555999273s) [noise=true , address=45.91.116.4:4967 , piece_id=O2BFLQCED5NTTO3AJ7ORLUQIFJTEEM3NGHLRH6PSXQU5T6JVQNJQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:126K4SnvjVs8h9Um58nMKgc6CGtPE99TKCAE5erxaKNguCETEWd, address:45.91.116.4:4967): protocol: rpc: dial tcp 45.91.116.4:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197695507 (266h49m43.556354148s) [noise=true , address=45.91.116.14:4207 , piece_id=ABU7RGDESBJ7HPXW5OXXRJ4K7CNGXIIH5CCDDJQ47NRMJJJS4HQQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:12faeeA3tF8s8WfxSwt75951poF8ZfeFUsEJNVXP22UeQCiBLT2, address:45.91.116.14:4207): protocol: rpc: dial tcp 45.91.116.14:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.197889209 (266h49m43.556547869s) [noise=true , address=77.90.31.11:4576 , piece_id=IGZ6Q5HFNP2KEJMPRO7YIA3RYOW7DHZUIXPH3NMGXBLQ4WTNTTXQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv, address:77.90.31.11:4576): protocol: read tcp 139.59.144.22:45408->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198091780 (266h49m43.556750444s) [noise=true , address=77.90.31.19:4848 , piece_id=B5F5OZI7GUS2AGGRVKU657PSYWJVVJT5FWACY5X57T73G2FMKEMQ, seg_pos={0 0}, upload=235275]: Done uploading piece. err: ecclient: upload failed (node:1EFoU2X25nensqiGn1NwTk1LcVDFK9yeJaGBm1nDTe8GqfGXwD, address:77.90.31.19:4848): protocol: rpc: dial tcp 77.90.31.19:4848: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:33:58.198131538 (266h49m43.556790181s) [seg_pos={0 0}, upload=235275]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:33:58.303167360 (266h49m43.661826002s) [upload=235275]: Done uploading segments...
Nov 22 12:33:58.303212446 (266h49m43.661871099s) [upload=235275]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 1m0.109740652s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..................................................X..XX..XX.] duration:
1.641882565s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:33:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.641910125s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................................X...XX...X] duration:
30.000712414s trace:
https://jaeger.dev.storj.io/trace/220a09ca74d9d1c8
FAIL (2025-11-22 12:33:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000760473s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...................................................X..XX..XX] duration:
5.001512501s trace:
https://jaeger.dev.storj.io/trace/5bcba4939824f50a
FAIL (2025-11-22 12:33:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001532196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....................................................X..XX..X] duration:
5.001608298s trace:
https://jaeger.dev.storj.io/trace/4b13d5610babdf7f
FAIL (2025-11-22 12:33:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001632845s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X..X...] duration:
2.283466969s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:32:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.283491565s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.265319193s trace:
https://jaeger.dev.storj.io/trace/7df908bc043b5ccb
FAIL (2025-11-22 12:32:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
VZBG53Q75GSQFZFKI7YLHQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.399196928 (266h46m32.757855587s) [noise=true , address=77.90.31.10:4539 , piece_id=FAD5NN7WENMXHWQBPTBUOX4CZUR53LSPPWIBHEAYDL4IDDZUWPWA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.399467946 (266h46m32.758126599s) [noise=true , address=145.79.227.11:4735 , piece_id=JYNOKF4UCBL3R6UFLRMPXPW4YHRHQ3TMCE5ZUNM7K35OVE54HNWA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.399742501 (266h46m32.758401156s) [noise=true , address=77.90.31.14:4664 , piece_id=7I5EQJYTVHDYP2TFKX6WJD3PAFXE3A2UZJU2CRNH3IXFNAX7SIWA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.400024013 (266h46m32.758682669s) [noise=true , address=45.91.116.9:4265 , piece_id=SVORNJRGRVMM6XP7L2WMOMKI74IKQL4OJ6XXHOAJSC3C6Y233YEA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.400696297 (266h46m32.759354956s) [noise=true , address=145.79.227.30:4764 , piece_id=EFWEZ4UTIEGXBH6AWPE5FNZNJH6GO54IEKQK4ULYTXBQYDQDCHDA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.401263680 (266h46m32.759922339s) [noise=true , address=77.90.31.19:4871 , piece_id=VNEGW3EEDISSLS3I325SW3WAPXAFXFUAFDCY6IGLZB2WDOD3UKLA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.401664817 (266h46m32.760323476s) [noise=true , address=145.79.227.28:4749 , piece_id=IV3H7OC5DZE5A6J33DSC5EO6XTW2NISPIZUUUS3YJIY6BXV6GCXA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.402099061 (266h46m32.760757715s) [noise=true , address=77.90.31.13:4654 , piece_id=TLAMYVHJF3IBYO7GRESALJVMQN44GRKNIZQGX2VC6PLDWNMKUIXA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.402543354 (266h46m32.761202026s) [noise=true , address=45.91.116.4:4961 , piece_id=IBSJVTN5KQ7MA5Q6VKS2NVMI7FK76NGPE3TIKNQ3XFHMJB7CL6BA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.402971181 (266h46m32.761629834s) [noise=true , address=77.90.31.11:4564 , piece_id=NF3WZ4JEER6EQ33WN35734T2JE57ZL427R5NTHNGHINAW2UDGQIA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.403438007 (266h46m32.762096661s) [noise=true , address=77.90.31.17:4777 , piece_id=LBFGDLF6ELRQHRWHCS3JGLQZHXLJDOZ7XZH347KLNES5R4WDKNBQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.403929218 (266h46m32.76258788s) [noise=true , address=77.90.31.15:4719 , piece_id=TZLL4H4DGYHZWKQWSAD7ZZMXRY67HNWX4YJ4LFWY5PGQWZ6ETA4A, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.404441396 (266h46m32.76310005s) [noise=true , address=45.91.116.14:4204 , piece_id=5BGKWC6RGBZSYNNGQCT5W6ZUIIONDZZPZIIK6WIN6OOQRCDTL3RQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.404912157 (266h46m32.763570813s) [noise=true , address=77.90.31.21:4902 , piece_id=BXQKAYFWOQ46AJ55XGW7AM5SRM3RAJLQ3KAURRVORJTI5CEAOYDA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.405447699 (266h46m32.764106353s) [noise=true , address=145.79.227.17:4979 , piece_id=EGSR5WLALRJ3FOKJN6WH5VTQQL2OSHZYRRD3PLZARWDMIHH3X2OQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.964298084 (266h46m33.322956739s) [upload=235239]: Next returned nil segment
Nov 22 12:30:47.964311203 (266h46m33.322969857s) [upload=235239]: Waiting for error group managing segments...
Nov 22 12:30:48.102703689 (266h46m33.461362342s) [noise=true , address=145.79.227.30:4764 , piece_id=EFWEZ4UTIEGXBH6AWPE5FNZNJH6GO54IEKQK4ULYTXBQYDQDCHDA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:30:48.103762545 (266h46m33.462421198s) [noise=true , address=145.79.227.28:4749 , piece_id=IV3H7OC5DZE5A6J33DSC5EO6XTW2NISPIZUUUS3YJIY6BXV6GCXA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:30:48.110843683 (266h46m33.469502348s) [noise=true , address=145.79.227.11:4735 , piece_id=JYNOKF4UCBL3R6UFLRMPXPW4YHRHQ3TMCE5ZUNM7K35OVE54HNWA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:30:48.113739397 (266h46m33.47239805s) [noise=true , address=145.79.227.17:4979 , piece_id=EGSR5WLALRJ3FOKJN6WH5VTQQL2OSHZYRRD3PLZARWDMIHH3X2OQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:31:35.089121027 (266h47m20.447779683s) [noise=true , address=45.91.116.14:4204 , piece_id=5BGKWC6RGBZSYNNGQCT5W6ZUIIONDZZPZIIK6WIN6OOQRCDTL3RQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:31:47.006941953 (266h47m32.365600612s) [noise=true , address=77.90.31.13:4654 , piece_id=TLAMYVHJF3IBYO7GRESALJVMQN44GRKNIZQGX2VC6PLDWNMKUIXA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007209585 (266h47m32.365868241s) [noise=true , address=45.91.116.4:4961 , piece_id=IBSJVTN5KQ7MA5Q6VKS2NVMI7FK76NGPE3TIKNQ3XFHMJB7CL6BA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007351613 (266h47m32.366010269s) [noise=true , address=77.90.31.17:4777 , piece_id=LBFGDLF6ELRQHRWHCS3JGLQZHXLJDOZ7XZH347KLNES5R4WDKNBQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007599427 (266h47m32.366258087s) [noise=true , address=77.90.31.15:4719 , piece_id=TZLL4H4DGYHZWKQWSAD7ZZMXRY67HNWX4YJ4LFWY5PGQWZ6ETA4A, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1qyhppjd6m6he7eHVZLuhK9BYKXq1DY3GuKBNfyb97JVaz9wyk, address:77.90.31.15:4719): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007722711 (266h47m32.36638137s) [noise=true , address=77.90.31.21:4902 , piece_id=BXQKAYFWOQ46AJ55XGW7AM5SRM3RAJLQ3KAURRVORJTI5CEAOYDA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008014205 (266h47m32.366672878s) [noise=true , address=77.90.31.19:4871 , piece_id=VNEGW3EEDISSLS3I325SW3WAPXAFXFUAFDCY6IGLZB2WDOD3UKLA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008176857 (266h47m32.366835522s) [noise=true , address=45.91.116.9:4265 , piece_id=SVORNJRGRVMM6XP7L2WMOMKI74IKQL4OJ6XXHOAJSC3C6Y233YEA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008506779 (266h47m32.367165438s) [noise=true , address=77.90.31.11:4564 , piece_id=NF3WZ4JEER6EQ33WN35734T2JE57ZL427R5NTHNGHINAW2UDGQIA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008634894 (266h47m32.367293553s) [noise=true , address=45.91.116.6:4002 , piece_id=SEQ2AS2KQGZOLYYTI3OVWW3EMLF2HBECP3MSP4DMRTN3LCTKCBKA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12FYX3Kj1Te1CZCEq81YVnKczjy9cy8e7RWzTJft2jwae6ZvaeY, address:45.91.116.6:4002): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008909787 (266h47m32.367568446s) [noise=true , address=45.91.116.7:4022 , piece_id=BOCUS2GYQKOAWD46VXLBUVZC6KYVXPVZBG53Q75GSQFZFKI7YLHQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi, address:45.91.116.7:4022): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.009426612 (266h47m32.368085266s) [noise=true , address=77.90.31.10:4539 , piece_id=FAD5NN7WENMXHWQBPTBUOX4CZUR53LSPPWIBHEAYDL4IDDZUWPWA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.009715169 (266h47m32.368373828s) [noise=true , address=77.90.31.12:4611 , piece_id=MMB45CNVIINFPZBAXBYDQ32KY7SIURIQX5IQFUH3MFME6HC32PBQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1r7qdqiUuUqsKnTXdC3S1Ek84rsCPtjPoXbPugVHWBzJezv4W2, address:77.90.31.12:4611): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.009840438 (266h47m32.368499085s) [noise=true , address=77.90.31.14:4664 , piece_id=7I5EQJYTVHDYP2TFKX6WJD3PAFXE3A2UZJU2CRNH3IXFNAX7SIWA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: write tcp 139.59.144.22:37380->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.009861732 (266h47m32.368520374s) [seg_pos={0 0}, upload=235239]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:31:47.125108267 (266h47m32.48376693s) [upload=235239]: Done uploading segments...
Nov 22 12:31:47.125166455 (266h47m32.483825114s) [upload=235239]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.265352975s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.....................................................X..XX..] duration:
1.582489412s trace:
https://jaeger.dev.storj.io/trace/16315206915c2d69
FAIL (2025-11-22 12:32:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.582549625s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X..X..] duration:
2.345954304s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:32:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.345997299s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................................X...XX...] duration:
2.892099672s trace:
https://jaeger.dev.storj.io/trace/10ea642e39e81020
FAIL (2025-11-22 12:32:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.892125751s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................................X...XX..] duration:
2.475102759s trace:
https://jaeger.dev.storj.io/trace/10ea642e39e81020
FAIL (2025-11-22 12:31:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.47513394s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.281494637s trace:
https://jaeger.dev.storj.io/trace/7df908bc043b5ccb
FAIL (2025-11-22 12:31:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
VZBG53Q75GSQFZFKI7YLHQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.399196928 (266h46m32.757855587s) [noise=true , address=77.90.31.10:4539 , piece_id=FAD5NN7WENMXHWQBPTBUOX4CZUR53LSPPWIBHEAYDL4IDDZUWPWA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.399467946 (266h46m32.758126599s) [noise=true , address=145.79.227.11:4735 , piece_id=JYNOKF4UCBL3R6UFLRMPXPW4YHRHQ3TMCE5ZUNM7K35OVE54HNWA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.399742501 (266h46m32.758401156s) [noise=true , address=77.90.31.14:4664 , piece_id=7I5EQJYTVHDYP2TFKX6WJD3PAFXE3A2UZJU2CRNH3IXFNAX7SIWA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.400024013 (266h46m32.758682669s) [noise=true , address=45.91.116.9:4265 , piece_id=SVORNJRGRVMM6XP7L2WMOMKI74IKQL4OJ6XXHOAJSC3C6Y233YEA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.400696297 (266h46m32.759354956s) [noise=true , address=145.79.227.30:4764 , piece_id=EFWEZ4UTIEGXBH6AWPE5FNZNJH6GO54IEKQK4ULYTXBQYDQDCHDA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.401263680 (266h46m32.759922339s) [noise=true , address=77.90.31.19:4871 , piece_id=VNEGW3EEDISSLS3I325SW3WAPXAFXFUAFDCY6IGLZB2WDOD3UKLA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.401664817 (266h46m32.760323476s) [noise=true , address=145.79.227.28:4749 , piece_id=IV3H7OC5DZE5A6J33DSC5EO6XTW2NISPIZUUUS3YJIY6BXV6GCXA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.402099061 (266h46m32.760757715s) [noise=true , address=77.90.31.13:4654 , piece_id=TLAMYVHJF3IBYO7GRESALJVMQN44GRKNIZQGX2VC6PLDWNMKUIXA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.402543354 (266h46m32.761202026s) [noise=true , address=45.91.116.4:4961 , piece_id=IBSJVTN5KQ7MA5Q6VKS2NVMI7FK76NGPE3TIKNQ3XFHMJB7CL6BA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.402971181 (266h46m32.761629834s) [noise=true , address=77.90.31.11:4564 , piece_id=NF3WZ4JEER6EQ33WN35734T2JE57ZL427R5NTHNGHINAW2UDGQIA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.403438007 (266h46m32.762096661s) [noise=true , address=77.90.31.17:4777 , piece_id=LBFGDLF6ELRQHRWHCS3JGLQZHXLJDOZ7XZH347KLNES5R4WDKNBQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.403929218 (266h46m32.76258788s) [noise=true , address=77.90.31.15:4719 , piece_id=TZLL4H4DGYHZWKQWSAD7ZZMXRY67HNWX4YJ4LFWY5PGQWZ6ETA4A, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.404441396 (266h46m32.76310005s) [noise=true , address=45.91.116.14:4204 , piece_id=5BGKWC6RGBZSYNNGQCT5W6ZUIIONDZZPZIIK6WIN6OOQRCDTL3RQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.404912157 (266h46m32.763570813s) [noise=true , address=77.90.31.21:4902 , piece_id=BXQKAYFWOQ46AJ55XGW7AM5SRM3RAJLQ3KAURRVORJTI5CEAOYDA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.405447699 (266h46m32.764106353s) [noise=true , address=145.79.227.17:4979 , piece_id=EGSR5WLALRJ3FOKJN6WH5VTQQL2OSHZYRRD3PLZARWDMIHH3X2OQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.964298084 (266h46m33.322956739s) [upload=235239]: Next returned nil segment
Nov 22 12:30:47.964311203 (266h46m33.322969857s) [upload=235239]: Waiting for error group managing segments...
Nov 22 12:30:48.102703689 (266h46m33.461362342s) [noise=true , address=145.79.227.30:4764 , piece_id=EFWEZ4UTIEGXBH6AWPE5FNZNJH6GO54IEKQK4ULYTXBQYDQDCHDA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:30:48.103762545 (266h46m33.462421198s) [noise=true , address=145.79.227.28:4749 , piece_id=IV3H7OC5DZE5A6J33DSC5EO6XTW2NISPIZUUUS3YJIY6BXV6GCXA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:30:48.110843683 (266h46m33.469502348s) [noise=true , address=145.79.227.11:4735 , piece_id=JYNOKF4UCBL3R6UFLRMPXPW4YHRHQ3TMCE5ZUNM7K35OVE54HNWA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:30:48.113739397 (266h46m33.47239805s) [noise=true , address=145.79.227.17:4979 , piece_id=EGSR5WLALRJ3FOKJN6WH5VTQQL2OSHZYRRD3PLZARWDMIHH3X2OQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:31:35.089121027 (266h47m20.447779683s) [noise=true , address=45.91.116.14:4204 , piece_id=5BGKWC6RGBZSYNNGQCT5W6ZUIIONDZZPZIIK6WIN6OOQRCDTL3RQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:31:47.006941953 (266h47m32.365600612s) [noise=true , address=77.90.31.13:4654 , piece_id=TLAMYVHJF3IBYO7GRESALJVMQN44GRKNIZQGX2VC6PLDWNMKUIXA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007209585 (266h47m32.365868241s) [noise=true , address=45.91.116.4:4961 , piece_id=IBSJVTN5KQ7MA5Q6VKS2NVMI7FK76NGPE3TIKNQ3XFHMJB7CL6BA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007351613 (266h47m32.366010269s) [noise=true , address=77.90.31.17:4777 , piece_id=LBFGDLF6ELRQHRWHCS3JGLQZHXLJDOZ7XZH347KLNES5R4WDKNBQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007599427 (266h47m32.366258087s) [noise=true , address=77.90.31.15:4719 , piece_id=TZLL4H4DGYHZWKQWSAD7ZZMXRY67HNWX4YJ4LFWY5PGQWZ6ETA4A, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1qyhppjd6m6he7eHVZLuhK9BYKXq1DY3GuKBNfyb97JVaz9wyk, address:77.90.31.15:4719): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007722711 (266h47m32.36638137s) [noise=true , address=77.90.31.21:4902 , piece_id=BXQKAYFWOQ46AJ55XGW7AM5SRM3RAJLQ3KAURRVORJTI5CEAOYDA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008014205 (266h47m32.366672878s) [noise=true , address=77.90.31.19:4871 , piece_id=VNEGW3EEDISSLS3I325SW3WAPXAFXFUAFDCY6IGLZB2WDOD3UKLA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008176857 (266h47m32.366835522s) [noise=true , address=45.91.116.9:4265 , piece_id=SVORNJRGRVMM6XP7L2WMOMKI74IKQL4OJ6XXHOAJSC3C6Y233YEA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008506779 (266h47m32.367165438s) [noise=true , address=77.90.31.11:4564 , piece_id=NF3WZ4JEER6EQ33WN35734T2JE57ZL427R5NTHNGHINAW2UDGQIA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008634894 (266h47m32.367293553s) [noise=true , address=45.91.116.6:4002 , piece_id=SEQ2AS2KQGZOLYYTI3OVWW3EMLF2HBECP3MSP4DMRTN3LCTKCBKA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12FYX3Kj1Te1CZCEq81YVnKczjy9cy8e7RWzTJft2jwae6ZvaeY, address:45.91.116.6:4002): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008909787 (266h47m32.367568446s) [noise=true , address=45.91.116.7:4022 , piece_id=BOCUS2GYQKOAWD46VXLBUVZC6KYVXPVZBG53Q75GSQFZFKI7YLHQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi, address:45.91.116.7:4022): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.009426612 (266h47m32.368085266s) [noise=true , address=77.90.31.10:4539 , piece_id=FAD5NN7WENMXHWQBPTBUOX4CZUR53LSPPWIBHEAYDL4IDDZUWPWA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.009715169 (266h47m32.368373828s) [noise=true , address=77.90.31.12:4611 , piece_id=MMB45CNVIINFPZBAXBYDQ32KY7SIURIQX5IQFUH3MFME6HC32PBQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1r7qdqiUuUqsKnTXdC3S1Ek84rsCPtjPoXbPugVHWBzJezv4W2, address:77.90.31.12:4611): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.009840438 (266h47m32.368499085s) [noise=true , address=77.90.31.14:4664 , piece_id=7I5EQJYTVHDYP2TFKX6WJD3PAFXE3A2UZJU2CRNH3IXFNAX7SIWA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: write tcp 139.59.144.22:37380->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.009861732 (266h47m32.368520374s) [seg_pos={0 0}, upload=235239]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:31:47.125108267 (266h47m32.48376693s) [upload=235239]: Done uploading segments...
Nov 22 12:31:47.125166455 (266h47m32.483825114s) [upload=235239]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 2.281510864s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.150444468s trace:
https://jaeger.dev.storj.io/trace/7df908bc043b5ccb
FAIL (2025-11-22 12:31:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 5)
======LOGS======
VZBG53Q75GSQFZFKI7YLHQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.399196928 (266h46m32.757855587s) [noise=true , address=77.90.31.10:4539 , piece_id=FAD5NN7WENMXHWQBPTBUOX4CZUR53LSPPWIBHEAYDL4IDDZUWPWA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.399467946 (266h46m32.758126599s) [noise=true , address=145.79.227.11:4735 , piece_id=JYNOKF4UCBL3R6UFLRMPXPW4YHRHQ3TMCE5ZUNM7K35OVE54HNWA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.399742501 (266h46m32.758401156s) [noise=true , address=77.90.31.14:4664 , piece_id=7I5EQJYTVHDYP2TFKX6WJD3PAFXE3A2UZJU2CRNH3IXFNAX7SIWA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.400024013 (266h46m32.758682669s) [noise=true , address=45.91.116.9:4265 , piece_id=SVORNJRGRVMM6XP7L2WMOMKI74IKQL4OJ6XXHOAJSC3C6Y233YEA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.400696297 (266h46m32.759354956s) [noise=true , address=145.79.227.30:4764 , piece_id=EFWEZ4UTIEGXBH6AWPE5FNZNJH6GO54IEKQK4ULYTXBQYDQDCHDA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.401263680 (266h46m32.759922339s) [noise=true , address=77.90.31.19:4871 , piece_id=VNEGW3EEDISSLS3I325SW3WAPXAFXFUAFDCY6IGLZB2WDOD3UKLA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.401664817 (266h46m32.760323476s) [noise=true , address=145.79.227.28:4749 , piece_id=IV3H7OC5DZE5A6J33DSC5EO6XTW2NISPIZUUUS3YJIY6BXV6GCXA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.402099061 (266h46m32.760757715s) [noise=true , address=77.90.31.13:4654 , piece_id=TLAMYVHJF3IBYO7GRESALJVMQN44GRKNIZQGX2VC6PLDWNMKUIXA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.402543354 (266h46m32.761202026s) [noise=true , address=45.91.116.4:4961 , piece_id=IBSJVTN5KQ7MA5Q6VKS2NVMI7FK76NGPE3TIKNQ3XFHMJB7CL6BA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.402971181 (266h46m32.761629834s) [noise=true , address=77.90.31.11:4564 , piece_id=NF3WZ4JEER6EQ33WN35734T2JE57ZL427R5NTHNGHINAW2UDGQIA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.403438007 (266h46m32.762096661s) [noise=true , address=77.90.31.17:4777 , piece_id=LBFGDLF6ELRQHRWHCS3JGLQZHXLJDOZ7XZH347KLNES5R4WDKNBQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.403929218 (266h46m32.76258788s) [noise=true , address=77.90.31.15:4719 , piece_id=TZLL4H4DGYHZWKQWSAD7ZZMXRY67HNWX4YJ4LFWY5PGQWZ6ETA4A, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.404441396 (266h46m32.76310005s) [noise=true , address=45.91.116.14:4204 , piece_id=5BGKWC6RGBZSYNNGQCT5W6ZUIIONDZZPZIIK6WIN6OOQRCDTL3RQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.404912157 (266h46m32.763570813s) [noise=true , address=77.90.31.21:4902 , piece_id=BXQKAYFWOQ46AJ55XGW7AM5SRM3RAJLQ3KAURRVORJTI5CEAOYDA, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.405447699 (266h46m32.764106353s) [noise=true , address=145.79.227.17:4979 , piece_id=EGSR5WLALRJ3FOKJN6WH5VTQQL2OSHZYRRD3PLZARWDMIHH3X2OQ, seg_pos={0 0}, upload=235239]: Uploading piece...
Nov 22 12:30:47.964298084 (266h46m33.322956739s) [upload=235239]: Next returned nil segment
Nov 22 12:30:47.964311203 (266h46m33.322969857s) [upload=235239]: Waiting for error group managing segments...
Nov 22 12:30:48.102703689 (266h46m33.461362342s) [noise=true , address=145.79.227.30:4764 , piece_id=EFWEZ4UTIEGXBH6AWPE5FNZNJH6GO54IEKQK4ULYTXBQYDQDCHDA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:30:48.103762545 (266h46m33.462421198s) [noise=true , address=145.79.227.28:4749 , piece_id=IV3H7OC5DZE5A6J33DSC5EO6XTW2NISPIZUUUS3YJIY6BXV6GCXA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:30:48.110843683 (266h46m33.469502348s) [noise=true , address=145.79.227.11:4735 , piece_id=JYNOKF4UCBL3R6UFLRMPXPW4YHRHQ3TMCE5ZUNM7K35OVE54HNWA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:30:48.113739397 (266h46m33.47239805s) [noise=true , address=145.79.227.17:4979 , piece_id=EGSR5WLALRJ3FOKJN6WH5VTQQL2OSHZYRRD3PLZARWDMIHH3X2OQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:31:35.089121027 (266h47m20.447779683s) [noise=true , address=45.91.116.14:4204 , piece_id=5BGKWC6RGBZSYNNGQCT5W6ZUIIONDZZPZIIK6WIN6OOQRCDTL3RQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: <nil>
Nov 22 12:31:47.006941953 (266h47m32.365600612s) [noise=true , address=77.90.31.13:4654 , piece_id=TLAMYVHJF3IBYO7GRESALJVMQN44GRKNIZQGX2VC6PLDWNMKUIXA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8, address:77.90.31.13:4654): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007209585 (266h47m32.365868241s) [noise=true , address=45.91.116.4:4961 , piece_id=IBSJVTN5KQ7MA5Q6VKS2NVMI7FK76NGPE3TIKNQ3XFHMJB7CL6BA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007351613 (266h47m32.366010269s) [noise=true , address=77.90.31.17:4777 , piece_id=LBFGDLF6ELRQHRWHCS3JGLQZHXLJDOZ7XZH347KLNES5R4WDKNBQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1DjhLCRQRkNECPQg8HG8kZcBn8jJmBZpNhdywxEK7ZUn1v19wM, address:77.90.31.17:4777): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007599427 (266h47m32.366258087s) [noise=true , address=77.90.31.15:4719 , piece_id=TZLL4H4DGYHZWKQWSAD7ZZMXRY67HNWX4YJ4LFWY5PGQWZ6ETA4A, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1qyhppjd6m6he7eHVZLuhK9BYKXq1DY3GuKBNfyb97JVaz9wyk, address:77.90.31.15:4719): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.007722711 (266h47m32.36638137s) [noise=true , address=77.90.31.21:4902 , piece_id=BXQKAYFWOQ46AJ55XGW7AM5SRM3RAJLQ3KAURRVORJTI5CEAOYDA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008014205 (266h47m32.366672878s) [noise=true , address=77.90.31.19:4871 , piece_id=VNEGW3EEDISSLS3I325SW3WAPXAFXFUAFDCY6IGLZB2WDOD3UKLA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008176857 (266h47m32.366835522s) [noise=true , address=45.91.116.9:4265 , piece_id=SVORNJRGRVMM6XP7L2WMOMKI74IKQL4OJ6XXHOAJSC3C6Y233YEA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008506779 (266h47m32.367165438s) [noise=true , address=77.90.31.11:4564 , piece_id=NF3WZ4JEER6EQ33WN35734T2JE57ZL427R5NTHNGHINAW2UDGQIA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9, address:77.90.31.11:4564): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008634894 (266h47m32.367293553s) [noise=true , address=45.91.116.6:4002 , piece_id=SEQ2AS2KQGZOLYYTI3OVWW3EMLF2HBECP3MSP4DMRTN3LCTKCBKA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12FYX3Kj1Te1CZCEq81YVnKczjy9cy8e7RWzTJft2jwae6ZvaeY, address:45.91.116.6:4002): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.008909787 (266h47m32.367568446s) [noise=true , address=45.91.116.7:4022 , piece_id=BOCUS2GYQKOAWD46VXLBUVZC6KYVXPVZBG53Q75GSQFZFKI7YLHQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12AsdD55hrnDtX47RL9vvS2nEeKt81AxYf3sgf5YAcTGggC6Rbi, address:45.91.116.7:4022): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.009426612 (266h47m32.368085266s) [noise=true , address=77.90.31.10:4539 , piece_id=FAD5NN7WENMXHWQBPTBUOX4CZUR53LSPPWIBHEAYDL4IDDZUWPWA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka, address:77.90.31.10:4539): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.009715169 (266h47m32.368373828s) [noise=true , address=77.90.31.12:4611 , piece_id=MMB45CNVIINFPZBAXBYDQ32KY7SIURIQX5IQFUH3MFME6HC32PBQ, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:1r7qdqiUuUqsKnTXdC3S1Ek84rsCPtjPoXbPugVHWBzJezv4W2, address:77.90.31.12:4611): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:47.009840438 (266h47m32.368499085s) [noise=true , address=77.90.31.14:4664 , piece_id=7I5EQJYTVHDYP2TFKX6WJD3PAFXE3A2UZJU2CRNH3IXFNAX7SIWA, seg_pos={0 0}, upload=235239]: Done uploading piece. err: ecclient: upload failed (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, address:77.90.31.14:4664): protocol: write tcp 139.59.144.22:37380->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:47.009861732 (266h47m32.368520374s) [seg_pos={0 0}, upload=235239]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil>
Nov 22 12:31:47.125108267 (266h47m32.48376693s) [upload=235239]: Done uploading segments...
Nov 22 12:31:47.125166455 (266h47m32.483825114s) [upload=235239]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5)
Duration: 1m0.150469018s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration:
2.401924614s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:31:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.401953041s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [......................................................X..XX.] duration:
1.564494878s trace:
https://jaeger.dev.storj.io/trace/16315206915c2d69
FAIL (2025-11-22 12:31:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.564515968s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................................X...XX.] duration:
4.198795369s trace:
https://jaeger.dev.storj.io/trace/10ea642e39e81020
FAIL (2025-11-22 12:31:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.198806255s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................................................X...XX] duration:
30.000659514s trace:
https://jaeger.dev.storj.io/trace/10ea642e39e81020
FAIL (2025-11-22 12:31:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000668896s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................X..X] duration:
30.10543742s trace:
https://jaeger.dev.storj.io/trace/5656a0c7e0b53cda
FAIL (2025-11-22 12:31:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:30:52.003787021 (266h46m37.362445675s) [upload=235241]: Starting upload
Nov 22 12:30:52.003885048 (266h46m37.362543729s) [upload=235241]: Done starting upload
Nov 22 12:30:52.003977160 (266h46m37.362635814s) [upload=235241]: Uploading segments...
Nov 22 12:30:52.004140730 (266h46m37.362799384s) [upload=235241]: Got next segment. Inline: false
Nov 22 12:30:52.004176073 (266h46m37.362834729s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:30:52.004181703 (266h46m37.362840356s) [upload=235241]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:30:52.113539293 (266h46m37.472197949s) [seg_pos={0 0}, upload=235241]: Begin upload segment...
Nov 22 12:30:52.113592796 (266h46m37.472251458s) [seg_pos={0 0}, upload=235241]: Done begin upload segment.
Nov 22 12:30:52.113622689 (266h46m37.472281343s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.113670993 (266h46m37.472329655s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114132816 (266h46m37.472791473s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114351061 (266h46m37.473009726s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114514284 (266h46m37.47317294s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114801466 (266h46m37.473460129s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.114835097 (266h46m37.473493753s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115174456 (266h46m37.47383311s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115233219 (266h46m37.473891879s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115559987 (266h46m37.474218641s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115737038 (266h46m37.474395697s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.115831971 (266h46m37.474490611s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116101068 (266h46m37.474759721s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116130009 (266h46m37.474788662s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116505131 (266h46m37.475163791s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.116936492 (266h46m37.47559515s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Uploading piece...
Nov 22 12:30:52.720203965 (266h46m38.078862618s) [upload=235241]: Next returned nil segment
Nov 22 12:30:52.720216190 (266h46m38.078874829s) [upload=235241]: Waiting for error group managing segments...
Nov 22 12:30:52.859097543 (266h46m38.2177562s) [noise=true , address=145.79.227.28:4726 , piece_id=EHG25AF4CZ3UMVOVG6TSX3TFAXMTFZND4INOLLX5IZUM3744FQQA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859631290 (266h46m38.218289947s) [noise=true , address=145.79.227.27:4691 , piece_id=HJV42J2TB2Y5SXVDTKJS265UQ34MAPMNAH2ZWH3NUXBWLXHWOOYQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.859726981 (266h46m38.21838564s) [noise=true , address=145.79.227.31:4799 , piece_id=6J5APHQR2STIABAKSUZQEBPSR6XIVVK2LXNDU6JDMIBCJRGT54GA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.861545214 (266h46m38.22020387s) [noise=true , address=145.79.227.10:4402 , piece_id=2QSJODIK6I6QYQXATBBLAL5SDGB7O3H5OO4GLRQAGUCAXHI337VA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.864774401 (266h46m38.223433064s) [noise=true , address=145.79.227.25:4619 , piece_id=4KKZOITVXGP7TQ5Q3WGKU5IOSY7CBDX4YHNKUD2TAFRFCNDDFWLA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.868389753 (266h46m38.22704841s) [noise=true , address=145.79.227.17:4970 , piece_id=VGCXEGCG2ZJSGHCPUW6TEJC33G3PT4OGK2WG6PCW6FBYUOH5Q6LA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.871585428 (266h46m38.230244094s) [noise=true , address=145.79.227.11:4730 , piece_id=A6MKVCMMOEDQCNVLZVLQE7727WRCTWSQESD5OTLYMEXXPGZXULYA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.873727410 (266h46m38.232386067s) [noise=true , address=145.79.227.22:4509 , piece_id=WZ556AW2TRVM3GC2DQEFDSBXOGTYGDBH424ZLBFOBB5NKAT3JUKA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.875573701 (266h46m38.234232366s) [noise=true , address=145.79.227.22:4510 , piece_id=GEQO4OVLPB3CWCYVOCAWMP7WUYVRIWNNICOUSTHHLKWXNQL46HLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.877964136 (266h46m38.236622776s) [noise=true , address=145.79.227.15:4875 , piece_id=RN4TLLBVGZQ2LZZES36O4XGTPSZLU4EYMA5VZ7DZBP46H342C64A, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:30:52.884853383 (266h46m38.243512048s) [noise=true , address=145.79.227.24:4581 , piece_id=7Y32DQ7BHOPIZWX7STPG2YHOWBY5Q3XW74S7D2U25POIKVKFYIUQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: <nil>
Nov 22 12:31:21.718054035 (266h47m7.076712695s) [noise=true , address=77.90.31.21:4877 , piece_id=AAKMO3BX7T6ISQFVZKNSL374X6PMRPBGWCMXW2NIAUILGWN2Y6WQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:44008->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718401995 (266h47m7.077060726s) [noise=true , address=77.90.31.18:4806 , piece_id=RSJWLFHBYMTJO47QF7ZK3ZE2GV7Z2B2T3NQDMH644VRFUQAJCYVA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, address:77.90.31.18:4806): protocol: rpc: dial tcp 77.90.31.18:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718466899 (266h47m7.077125546s) [noise=true , address=45.91.116.4:4939 , piece_id=CY72GWLHSUZJ6JL2O57SYBNYYTKC4CMT3EKX4NWFE3JWWXB7YVLQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:49860->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718639920 (266h47m7.07729858s) [noise=true , address=45.91.116.6:4975 , piece_id=CVSCYXLALCDU5O4NPWK4T73Q5DZFFM4YA6EZMNGSCYRRYLOGAGSA, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:31:21.718670710 (266h47m7.077329363s) [noise=true , address=45.91.116.11:4117 , piece_id=VJ2G5TGE46SVXX574ZGMRQUER6CJUYDHB47BNVWNPBIFDFFNKZBQ, seg_pos={0 0}, upload=235241]: Done uploading piece. err: ecclient: upload failed (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, address:45.91.116.11:4117): protocol: read tcp 139.59.144.22:40600->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:31:21.718696534 (266h47m7.077355177s) [seg_pos={0 0}, upload=235241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:31:21.821464823 (266h47m7.180123476s) [upload=235241]: Done uploading segments...
Nov 22 12:31:21.821497872 (266h47m7.180156526s) [upload=235241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.105457164s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.......................................................X..XX] duration:
5.001414066s trace:
https://jaeger.dev.storj.io/trace/16315206915c2d69
FAIL (2025-11-22 12:31:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.0014339s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X...X] duration:
30.001335589s trace:
https://jaeger.dev.storj.io/trace/7f6b1b7d67f39874
FAIL (2025-11-22 12:30:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001357619s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................................................X..X] duration:
5.001122789s trace:
https://jaeger.dev.storj.io/trace/6e3e1ea26f5b2ecf
FAIL (2025-11-22 12:30:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001165929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.246020807s trace:
https://jaeger.dev.storj.io/trace/51125b0f567b79ac
FAIL (2025-11-22 12:30:17 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:01.919416928 (266h43m47.278075583s) [upload=235200]: Starting upload
Nov 22 12:28:01.919480818 (266h43m47.278139473s) [upload=235200]: Done starting upload
Nov 22 12:28:01.919615272 (266h43m47.278273952s) [upload=235200]: Uploading segments...
Nov 22 12:28:01.920129622 (266h43m47.278788286s) [upload=235200]: Got next segment. Inline: false
Nov 22 12:28:01.920467007 (266h43m47.279125669s) [upload=235200]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:01.920490083 (266h43m47.279148738s) [upload=235200]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:02.030268044 (266h43m47.388926702s) [seg_pos={0 0}, upload=235200]: Begin upload segment...
Nov 22 12:28:02.030369022 (266h43m47.389027683s) [noise=true , address=145.79.227.20:4439 , piece_id=A3QCT2DI7K5ZLH76TCSZNKM5AWBQS4XOQKOWBCKW276IDPYPAB4Q, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.034078188 (266h43m47.392736845s) [seg_pos={0 0}, upload=235200]: Done begin upload segment.
Nov 22 12:28:02.034174332 (266h43m47.392833s) [noise=true , address=145.79.227.11:4730 , piece_id=VPOJUOE426WFMOCZQHMDTVUPRXNZJJJE3BBR5PGEKLXZGO2DB6EQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.034951433 (266h43m47.393610102s) [noise=true , address=45.91.116.12:4225 , piece_id=TH274VOPTSMGHPZYXRZ6LYRALHQIEEODGN2YLPLZ2NJSY2A5EGXQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.036810303 (266h43m47.395468971s) [noise=true , address=77.90.31.13:4628 , piece_id=N7U7DJ63UWL463WOIKQ4B33BQX3PAANH7DEXXQ6JRBK23VMNOG5A, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.039091569 (266h43m47.39775026s) [noise=true , address=145.79.227.21:4475 , piece_id=D3JAG7JOALQZCFKSRQYNRVSQFU6X47LC64Y65BHTE4BN7E7YEQYQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.039359308 (266h43m47.398017977s) [noise=true , address=145.79.227.6:4258 , piece_id=EN47NLQULZMBOPCZUPKBEA5KIIYNOU6DH4P3FFGJNO3NJXA5XJYA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.041951754 (266h43m47.400610415s) [noise=true , address=145.79.227.14:4837 , piece_id=SG67SGG3RYZGRXADNVM352ONMONYJABAOL5NXECDFYD7KYHIQZNA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.043248768 (266h43m47.401907433s) [noise=true , address=145.79.227.25:4619 , piece_id=OBEW4APNLTAIQZ4NVJW65IFP3QJU2WWC2TGW27XFVWV7BFVQE6WQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.044121915 (266h43m47.402780583s) [noise=true , address=145.79.227.12:4767 , piece_id=74PSXYTWYLWMUBGSMD6TU6ECYWZGTWLUOBAZFZZNW3ZIAKO3Q4NQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.045770844 (266h43m47.404429565s) [noise=true , address=145.79.227.26:4653 , piece_id=5NHURGUQI64KD2W7XOZ3MQ5XSC4LKOWH2FMB6PLZGS5AJ4PIIVWQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.047043815 (266h43m47.405702483s) [noise=true , address=145.79.227.22:4510 , piece_id=UKZITHRBUE6XSTNZ75QPA6CXM47J5VA4JTVVLN2YEPP57GZEFUHQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.049880744 (266h43m47.408539411s) [noise=true , address=77.90.31.21:4877 , piece_id=EAZBORRL2KEJUPA4UBPA6WTYJ3DDKEWOJBRCGV4DUSZLXMC2MGSQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.051296178 (266h43m47.409954842s) [noise=true , address=145.79.227.17:4969 , piece_id=PDDQSVNS5SVL53IETHJT3ISNM4YMZKMFDIPA6CKGAF7AAX2R7SPA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.057653012 (266h43m47.41631168s) [noise=true , address=45.91.116.9:4263 , piece_id=VAKRYSB2XYX3NCJHDZSU74MSERC5476BAGT6EWRXMJAEMIKMUVHQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.059231071 (266h43m47.41788974s) [noise=true , address=45.91.116.8:4047 , piece_id=PJTK7WLEKPQIXYPZBU3WBTO5DGXCVFKLFR2VEZKAT4FGM7G7BBGA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.062711176 (266h43m47.421369844s) [noise=true , address=145.79.227.10:4403 , piece_id=YF45JGLWYZ7NEGLJRR2ZFS7YV5VQIUKD7XSJLKLVQY3YJCJ2KZIA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.659147463 (266h43m48.017806127s) [upload=235200]: Next returned nil segment
Nov 22 12:28:02.659177752 (266h43m48.017836418s) [upload=235200]: Waiting for error group managing segments...
Nov 22 12:28:02.802314837 (266h43m48.160973504s) [noise=true , address=145.79.227.14:4837 , piece_id=SG67SGG3RYZGRXADNVM352ONMONYJABAOL5NXECDFYD7KYHIQZNA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.812545911 (266h43m48.171204567s) [noise=true , address=145.79.227.12:4767 , piece_id=74PSXYTWYLWMUBGSMD6TU6ECYWZGTWLUOBAZFZZNW3ZIAKO3Q4NQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.812946565 (266h43m48.171605222s) [noise=true , address=145.79.227.11:4730 , piece_id=VPOJUOE426WFMOCZQHMDTVUPRXNZJJJE3BBR5PGEKLXZGO2DB6EQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.813428679 (266h43m48.172087335s) [noise=true , address=145.79.227.10:4403 , piece_id=YF45JGLWYZ7NEGLJRR2ZFS7YV5VQIUKD7XSJLKLVQY3YJCJ2KZIA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.813854689 (266h43m48.172513344s) [noise=true , address=145.79.227.22:4510 , piece_id=UKZITHRBUE6XSTNZ75QPA6CXM47J5VA4JTVVLN2YEPP57GZEFUHQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.814320318 (266h43m48.172978975s) [noise=true , address=145.79.227.20:4439 , piece_id=A3QCT2DI7K5ZLH76TCSZNKM5AWBQS4XOQKOWBCKW276IDPYPAB4Q, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.814717265 (266h43m48.173375919s) [noise=true , address=145.79.227.6:4258 , piece_id=EN47NLQULZMBOPCZUPKBEA5KIIYNOU6DH4P3FFGJNO3NJXA5XJYA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.815557597 (266h43m48.174216256s) [noise=true , address=145.79.227.25:4619 , piece_id=OBEW4APNLTAIQZ4NVJW65IFP3QJU2WWC2TGW27XFVWV7BFVQE6WQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.816139191 (266h43m48.174797847s) [noise=true , address=145.79.227.26:4653 , piece_id=5NHURGUQI64KD2W7XOZ3MQ5XSC4LKOWH2FMB6PLZGS5AJ4PIIVWQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.818536673 (266h43m48.177195346s) [noise=true , address=145.79.227.17:4969 , piece_id=PDDQSVNS5SVL53IETHJT3ISNM4YMZKMFDIPA6CKGAF7AAX2R7SPA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.823862626 (266h43m48.182521287s) [noise=true , address=145.79.227.21:4475 , piece_id=D3JAG7JOALQZCFKSRQYNRVSQFU6X47LC64Y65BHTE4BN7E7YEQYQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:31.647513820 (266h44m17.006172483s) [noise=true , address=77.90.31.21:4877 , piece_id=EAZBORRL2KEJUPA4UBPA6WTYJ3DDKEWOJBRCGV4DUSZLXMC2MGSQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:45714->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.647682296 (266h44m17.006340952s) [noise=true , address=45.91.116.12:4225 , piece_id=TH274VOPTSMGHPZYXRZ6LYRALHQIEEODGN2YLPLZ2NJSY2A5EGXQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:28:31.647783194 (266h44m17.00644184s) [noise=true , address=45.91.116.9:4263 , piece_id=VAKRYSB2XYX3NCJHDZSU74MSERC5476BAGT6EWRXMJAEMIKMUVHQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:54566->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.647933048 (266h44m17.006591707s) [noise=true , address=77.90.31.13:4628 , piece_id=N7U7DJ63UWL463WOIKQ4B33BQX3PAANH7DEXXQ6JRBK23VMNOG5A, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.648006034 (266h44m17.006664691s) [noise=true , address=45.91.116.8:4047 , piece_id=PJTK7WLEKPQIXYPZBU3WBTO5DGXCVFKLFR2VEZKAT4FGM7G7BBGA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:49642->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.648034473 (266h44m17.006693117s) [seg_pos={0 0}, upload=235200]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:31.747529469 (266h44m17.106188126s) [upload=235200]: Done uploading segments...
Nov 22 12:28:31.747571263 (266h44m17.106229924s) [upload=235200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.246045966s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.625023082s trace:
https://jaeger.dev.storj.io/trace/17892cac9b103a92
FAIL (2025-11-22 12:29:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.625051566s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
2.057261793s trace:
https://jaeger.dev.storj.io/trace/74063d5209f81b22
FAIL (2025-11-22 12:29:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.05728624s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.572232764s trace:
https://jaeger.dev.storj.io/trace/17892cac9b103a92
FAIL (2025-11-22 12:29:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.572262574s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
1.587648402s trace:
https://jaeger.dev.storj.io/trace/74063d5209f81b22
FAIL (2025-11-22 12:29:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.587693666s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.304799645s trace:
https://jaeger.dev.storj.io/trace/51125b0f567b79ac
FAIL (2025-11-22 12:29:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:01.919416928 (266h43m47.278075583s) [upload=235200]: Starting upload
Nov 22 12:28:01.919480818 (266h43m47.278139473s) [upload=235200]: Done starting upload
Nov 22 12:28:01.919615272 (266h43m47.278273952s) [upload=235200]: Uploading segments...
Nov 22 12:28:01.920129622 (266h43m47.278788286s) [upload=235200]: Got next segment. Inline: false
Nov 22 12:28:01.920467007 (266h43m47.279125669s) [upload=235200]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:01.920490083 (266h43m47.279148738s) [upload=235200]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:02.030268044 (266h43m47.388926702s) [seg_pos={0 0}, upload=235200]: Begin upload segment...
Nov 22 12:28:02.030369022 (266h43m47.389027683s) [noise=true , address=145.79.227.20:4439 , piece_id=A3QCT2DI7K5ZLH76TCSZNKM5AWBQS4XOQKOWBCKW276IDPYPAB4Q, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.034078188 (266h43m47.392736845s) [seg_pos={0 0}, upload=235200]: Done begin upload segment.
Nov 22 12:28:02.034174332 (266h43m47.392833s) [noise=true , address=145.79.227.11:4730 , piece_id=VPOJUOE426WFMOCZQHMDTVUPRXNZJJJE3BBR5PGEKLXZGO2DB6EQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.034951433 (266h43m47.393610102s) [noise=true , address=45.91.116.12:4225 , piece_id=TH274VOPTSMGHPZYXRZ6LYRALHQIEEODGN2YLPLZ2NJSY2A5EGXQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.036810303 (266h43m47.395468971s) [noise=true , address=77.90.31.13:4628 , piece_id=N7U7DJ63UWL463WOIKQ4B33BQX3PAANH7DEXXQ6JRBK23VMNOG5A, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.039091569 (266h43m47.39775026s) [noise=true , address=145.79.227.21:4475 , piece_id=D3JAG7JOALQZCFKSRQYNRVSQFU6X47LC64Y65BHTE4BN7E7YEQYQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.039359308 (266h43m47.398017977s) [noise=true , address=145.79.227.6:4258 , piece_id=EN47NLQULZMBOPCZUPKBEA5KIIYNOU6DH4P3FFGJNO3NJXA5XJYA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.041951754 (266h43m47.400610415s) [noise=true , address=145.79.227.14:4837 , piece_id=SG67SGG3RYZGRXADNVM352ONMONYJABAOL5NXECDFYD7KYHIQZNA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.043248768 (266h43m47.401907433s) [noise=true , address=145.79.227.25:4619 , piece_id=OBEW4APNLTAIQZ4NVJW65IFP3QJU2WWC2TGW27XFVWV7BFVQE6WQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.044121915 (266h43m47.402780583s) [noise=true , address=145.79.227.12:4767 , piece_id=74PSXYTWYLWMUBGSMD6TU6ECYWZGTWLUOBAZFZZNW3ZIAKO3Q4NQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.045770844 (266h43m47.404429565s) [noise=true , address=145.79.227.26:4653 , piece_id=5NHURGUQI64KD2W7XOZ3MQ5XSC4LKOWH2FMB6PLZGS5AJ4PIIVWQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.047043815 (266h43m47.405702483s) [noise=true , address=145.79.227.22:4510 , piece_id=UKZITHRBUE6XSTNZ75QPA6CXM47J5VA4JTVVLN2YEPP57GZEFUHQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.049880744 (266h43m47.408539411s) [noise=true , address=77.90.31.21:4877 , piece_id=EAZBORRL2KEJUPA4UBPA6WTYJ3DDKEWOJBRCGV4DUSZLXMC2MGSQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.051296178 (266h43m47.409954842s) [noise=true , address=145.79.227.17:4969 , piece_id=PDDQSVNS5SVL53IETHJT3ISNM4YMZKMFDIPA6CKGAF7AAX2R7SPA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.057653012 (266h43m47.41631168s) [noise=true , address=45.91.116.9:4263 , piece_id=VAKRYSB2XYX3NCJHDZSU74MSERC5476BAGT6EWRXMJAEMIKMUVHQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.059231071 (266h43m47.41788974s) [noise=true , address=45.91.116.8:4047 , piece_id=PJTK7WLEKPQIXYPZBU3WBTO5DGXCVFKLFR2VEZKAT4FGM7G7BBGA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.062711176 (266h43m47.421369844s) [noise=true , address=145.79.227.10:4403 , piece_id=YF45JGLWYZ7NEGLJRR2ZFS7YV5VQIUKD7XSJLKLVQY3YJCJ2KZIA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.659147463 (266h43m48.017806127s) [upload=235200]: Next returned nil segment
Nov 22 12:28:02.659177752 (266h43m48.017836418s) [upload=235200]: Waiting for error group managing segments...
Nov 22 12:28:02.802314837 (266h43m48.160973504s) [noise=true , address=145.79.227.14:4837 , piece_id=SG67SGG3RYZGRXADNVM352ONMONYJABAOL5NXECDFYD7KYHIQZNA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.812545911 (266h43m48.171204567s) [noise=true , address=145.79.227.12:4767 , piece_id=74PSXYTWYLWMUBGSMD6TU6ECYWZGTWLUOBAZFZZNW3ZIAKO3Q4NQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.812946565 (266h43m48.171605222s) [noise=true , address=145.79.227.11:4730 , piece_id=VPOJUOE426WFMOCZQHMDTVUPRXNZJJJE3BBR5PGEKLXZGO2DB6EQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.813428679 (266h43m48.172087335s) [noise=true , address=145.79.227.10:4403 , piece_id=YF45JGLWYZ7NEGLJRR2ZFS7YV5VQIUKD7XSJLKLVQY3YJCJ2KZIA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.813854689 (266h43m48.172513344s) [noise=true , address=145.79.227.22:4510 , piece_id=UKZITHRBUE6XSTNZ75QPA6CXM47J5VA4JTVVLN2YEPP57GZEFUHQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.814320318 (266h43m48.172978975s) [noise=true , address=145.79.227.20:4439 , piece_id=A3QCT2DI7K5ZLH76TCSZNKM5AWBQS4XOQKOWBCKW276IDPYPAB4Q, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.814717265 (266h43m48.173375919s) [noise=true , address=145.79.227.6:4258 , piece_id=EN47NLQULZMBOPCZUPKBEA5KIIYNOU6DH4P3FFGJNO3NJXA5XJYA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.815557597 (266h43m48.174216256s) [noise=true , address=145.79.227.25:4619 , piece_id=OBEW4APNLTAIQZ4NVJW65IFP3QJU2WWC2TGW27XFVWV7BFVQE6WQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.816139191 (266h43m48.174797847s) [noise=true , address=145.79.227.26:4653 , piece_id=5NHURGUQI64KD2W7XOZ3MQ5XSC4LKOWH2FMB6PLZGS5AJ4PIIVWQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.818536673 (266h43m48.177195346s) [noise=true , address=145.79.227.17:4969 , piece_id=PDDQSVNS5SVL53IETHJT3ISNM4YMZKMFDIPA6CKGAF7AAX2R7SPA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.823862626 (266h43m48.182521287s) [noise=true , address=145.79.227.21:4475 , piece_id=D3JAG7JOALQZCFKSRQYNRVSQFU6X47LC64Y65BHTE4BN7E7YEQYQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:31.647513820 (266h44m17.006172483s) [noise=true , address=77.90.31.21:4877 , piece_id=EAZBORRL2KEJUPA4UBPA6WTYJ3DDKEWOJBRCGV4DUSZLXMC2MGSQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:45714->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.647682296 (266h44m17.006340952s) [noise=true , address=45.91.116.12:4225 , piece_id=TH274VOPTSMGHPZYXRZ6LYRALHQIEEODGN2YLPLZ2NJSY2A5EGXQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:28:31.647783194 (266h44m17.00644184s) [noise=true , address=45.91.116.9:4263 , piece_id=VAKRYSB2XYX3NCJHDZSU74MSERC5476BAGT6EWRXMJAEMIKMUVHQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:54566->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.647933048 (266h44m17.006591707s) [noise=true , address=77.90.31.13:4628 , piece_id=N7U7DJ63UWL463WOIKQ4B33BQX3PAANH7DEXXQ6JRBK23VMNOG5A, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.648006034 (266h44m17.006664691s) [noise=true , address=45.91.116.8:4047 , piece_id=PJTK7WLEKPQIXYPZBU3WBTO5DGXCVFKLFR2VEZKAT4FGM7G7BBGA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:49642->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.648034473 (266h44m17.006693117s) [seg_pos={0 0}, upload=235200]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:31.747529469 (266h44m17.106188126s) [upload=235200]: Done uploading segments...
Nov 22 12:28:31.747571263 (266h44m17.106229924s) [upload=235200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 2.304823829s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.969446738s trace:
https://jaeger.dev.storj.io/trace/17892cac9b103a92
FAIL (2025-11-22 12:28:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.969495383s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001510507s trace:
https://jaeger.dev.storj.io/trace/17892cac9b103a92
FAIL (2025-11-22 12:28:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001529333s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.001900008s trace:
https://jaeger.dev.storj.io/trace/74063d5209f81b22
FAIL (2025-11-22 12:28:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001920882s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.101659202s trace:
https://jaeger.dev.storj.io/trace/51125b0f567b79ac
FAIL (2025-11-22 12:28:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:01.919416928 (266h43m47.278075583s) [upload=235200]: Starting upload
Nov 22 12:28:01.919480818 (266h43m47.278139473s) [upload=235200]: Done starting upload
Nov 22 12:28:01.919615272 (266h43m47.278273952s) [upload=235200]: Uploading segments...
Nov 22 12:28:01.920129622 (266h43m47.278788286s) [upload=235200]: Got next segment. Inline: false
Nov 22 12:28:01.920467007 (266h43m47.279125669s) [upload=235200]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:01.920490083 (266h43m47.279148738s) [upload=235200]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:02.030268044 (266h43m47.388926702s) [seg_pos={0 0}, upload=235200]: Begin upload segment...
Nov 22 12:28:02.030369022 (266h43m47.389027683s) [noise=true , address=145.79.227.20:4439 , piece_id=A3QCT2DI7K5ZLH76TCSZNKM5AWBQS4XOQKOWBCKW276IDPYPAB4Q, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.034078188 (266h43m47.392736845s) [seg_pos={0 0}, upload=235200]: Done begin upload segment.
Nov 22 12:28:02.034174332 (266h43m47.392833s) [noise=true , address=145.79.227.11:4730 , piece_id=VPOJUOE426WFMOCZQHMDTVUPRXNZJJJE3BBR5PGEKLXZGO2DB6EQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.034951433 (266h43m47.393610102s) [noise=true , address=45.91.116.12:4225 , piece_id=TH274VOPTSMGHPZYXRZ6LYRALHQIEEODGN2YLPLZ2NJSY2A5EGXQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.036810303 (266h43m47.395468971s) [noise=true , address=77.90.31.13:4628 , piece_id=N7U7DJ63UWL463WOIKQ4B33BQX3PAANH7DEXXQ6JRBK23VMNOG5A, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.039091569 (266h43m47.39775026s) [noise=true , address=145.79.227.21:4475 , piece_id=D3JAG7JOALQZCFKSRQYNRVSQFU6X47LC64Y65BHTE4BN7E7YEQYQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.039359308 (266h43m47.398017977s) [noise=true , address=145.79.227.6:4258 , piece_id=EN47NLQULZMBOPCZUPKBEA5KIIYNOU6DH4P3FFGJNO3NJXA5XJYA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.041951754 (266h43m47.400610415s) [noise=true , address=145.79.227.14:4837 , piece_id=SG67SGG3RYZGRXADNVM352ONMONYJABAOL5NXECDFYD7KYHIQZNA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.043248768 (266h43m47.401907433s) [noise=true , address=145.79.227.25:4619 , piece_id=OBEW4APNLTAIQZ4NVJW65IFP3QJU2WWC2TGW27XFVWV7BFVQE6WQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.044121915 (266h43m47.402780583s) [noise=true , address=145.79.227.12:4767 , piece_id=74PSXYTWYLWMUBGSMD6TU6ECYWZGTWLUOBAZFZZNW3ZIAKO3Q4NQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.045770844 (266h43m47.404429565s) [noise=true , address=145.79.227.26:4653 , piece_id=5NHURGUQI64KD2W7XOZ3MQ5XSC4LKOWH2FMB6PLZGS5AJ4PIIVWQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.047043815 (266h43m47.405702483s) [noise=true , address=145.79.227.22:4510 , piece_id=UKZITHRBUE6XSTNZ75QPA6CXM47J5VA4JTVVLN2YEPP57GZEFUHQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.049880744 (266h43m47.408539411s) [noise=true , address=77.90.31.21:4877 , piece_id=EAZBORRL2KEJUPA4UBPA6WTYJ3DDKEWOJBRCGV4DUSZLXMC2MGSQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.051296178 (266h43m47.409954842s) [noise=true , address=145.79.227.17:4969 , piece_id=PDDQSVNS5SVL53IETHJT3ISNM4YMZKMFDIPA6CKGAF7AAX2R7SPA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.057653012 (266h43m47.41631168s) [noise=true , address=45.91.116.9:4263 , piece_id=VAKRYSB2XYX3NCJHDZSU74MSERC5476BAGT6EWRXMJAEMIKMUVHQ, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.059231071 (266h43m47.41788974s) [noise=true , address=45.91.116.8:4047 , piece_id=PJTK7WLEKPQIXYPZBU3WBTO5DGXCVFKLFR2VEZKAT4FGM7G7BBGA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.062711176 (266h43m47.421369844s) [noise=true , address=145.79.227.10:4403 , piece_id=YF45JGLWYZ7NEGLJRR2ZFS7YV5VQIUKD7XSJLKLVQY3YJCJ2KZIA, seg_pos={0 0}, upload=235200]: Uploading piece...
Nov 22 12:28:02.659147463 (266h43m48.017806127s) [upload=235200]: Next returned nil segment
Nov 22 12:28:02.659177752 (266h43m48.017836418s) [upload=235200]: Waiting for error group managing segments...
Nov 22 12:28:02.802314837 (266h43m48.160973504s) [noise=true , address=145.79.227.14:4837 , piece_id=SG67SGG3RYZGRXADNVM352ONMONYJABAOL5NXECDFYD7KYHIQZNA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.812545911 (266h43m48.171204567s) [noise=true , address=145.79.227.12:4767 , piece_id=74PSXYTWYLWMUBGSMD6TU6ECYWZGTWLUOBAZFZZNW3ZIAKO3Q4NQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.812946565 (266h43m48.171605222s) [noise=true , address=145.79.227.11:4730 , piece_id=VPOJUOE426WFMOCZQHMDTVUPRXNZJJJE3BBR5PGEKLXZGO2DB6EQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.813428679 (266h43m48.172087335s) [noise=true , address=145.79.227.10:4403 , piece_id=YF45JGLWYZ7NEGLJRR2ZFS7YV5VQIUKD7XSJLKLVQY3YJCJ2KZIA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.813854689 (266h43m48.172513344s) [noise=true , address=145.79.227.22:4510 , piece_id=UKZITHRBUE6XSTNZ75QPA6CXM47J5VA4JTVVLN2YEPP57GZEFUHQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.814320318 (266h43m48.172978975s) [noise=true , address=145.79.227.20:4439 , piece_id=A3QCT2DI7K5ZLH76TCSZNKM5AWBQS4XOQKOWBCKW276IDPYPAB4Q, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.814717265 (266h43m48.173375919s) [noise=true , address=145.79.227.6:4258 , piece_id=EN47NLQULZMBOPCZUPKBEA5KIIYNOU6DH4P3FFGJNO3NJXA5XJYA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.815557597 (266h43m48.174216256s) [noise=true , address=145.79.227.25:4619 , piece_id=OBEW4APNLTAIQZ4NVJW65IFP3QJU2WWC2TGW27XFVWV7BFVQE6WQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.816139191 (266h43m48.174797847s) [noise=true , address=145.79.227.26:4653 , piece_id=5NHURGUQI64KD2W7XOZ3MQ5XSC4LKOWH2FMB6PLZGS5AJ4PIIVWQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.818536673 (266h43m48.177195346s) [noise=true , address=145.79.227.17:4969 , piece_id=PDDQSVNS5SVL53IETHJT3ISNM4YMZKMFDIPA6CKGAF7AAX2R7SPA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:02.823862626 (266h43m48.182521287s) [noise=true , address=145.79.227.21:4475 , piece_id=D3JAG7JOALQZCFKSRQYNRVSQFU6X47LC64Y65BHTE4BN7E7YEQYQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: <nil>
Nov 22 12:28:31.647513820 (266h44m17.006172483s) [noise=true , address=77.90.31.21:4877 , piece_id=EAZBORRL2KEJUPA4UBPA6WTYJ3DDKEWOJBRCGV4DUSZLXMC2MGSQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:45714->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.647682296 (266h44m17.006340952s) [noise=true , address=45.91.116.12:4225 , piece_id=TH274VOPTSMGHPZYXRZ6LYRALHQIEEODGN2YLPLZ2NJSY2A5EGXQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: context deadline exceeded; context deadline exceeded
Nov 22 12:28:31.647783194 (266h44m17.00644184s) [noise=true , address=45.91.116.9:4263 , piece_id=VAKRYSB2XYX3NCJHDZSU74MSERC5476BAGT6EWRXMJAEMIKMUVHQ, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 139.59.144.22:54566->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.647933048 (266h44m17.006591707s) [noise=true , address=77.90.31.13:4628 , piece_id=N7U7DJ63UWL463WOIKQ4B33BQX3PAANH7DEXXQ6JRBK23VMNOG5A, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.648006034 (266h44m17.006664691s) [noise=true , address=45.91.116.8:4047 , piece_id=PJTK7WLEKPQIXYPZBU3WBTO5DGXCVFKLFR2VEZKAT4FGM7G7BBGA, seg_pos={0 0}, upload=235200]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 139.59.144.22:49642->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:31.648034473 (266h44m17.006693117s) [seg_pos={0 0}, upload=235200]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:31.747529469 (266h44m17.106188126s) [upload=235200]: Done uploading segments...
Nov 22 12:28:31.747571263 (266h44m17.106229924s) [upload=235200]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.101688169s
core
uplink
us-select-1
|