Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 51 out of 60 is failed (threshold=3) [........XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.108910929s trace: https://jaeger.dev.storj.io/trace/65cfb2ba665083bf FAIL (2026-04-18 01:18:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== d=64GZH3O7QOXZSKTMBAWDQS5D4ZCVHXNVZWZZMSJMPPL4MGED545Q, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.257116837 (249h0m36.851228087s) [noise=true , address=145.79.227.21:4481 , piece_id=NF2TUTM6NPV747EU4NJY7HUKCUBAFJA3TR55J6P256OBCQ2WDOVQ, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.257558966 (249h0m36.851670217s) [noise=true , address=167.17.72.68:4672 , piece_id=EGYVPZ46GNAPKJHSFSBXRGW75SJUMLOS3BVNO33KVZJSUS564TVQ, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.257720838 (249h0m36.851832086s) [noise=true , address=145.79.227.11:4754 , piece_id=GSMWQWGDHUQYMOKMKKVLQTNHHQ6XXCN7TNQTG7V5KBKQQNC5LC3Q, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.258245259 (249h0m36.85235651s) [noise=true , address=45.91.116.10:4115 , piece_id=MSFHMO2JD5BOQXBGG3OWHCHLRFABDWIIPEXSB7YPZ3VRGBVJQYIQ, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.258286667 (249h0m36.852397912s) [noise=true , address=145.79.227.44:4203 , piece_id=FVMY23ZG6QQKMKIIS7BEL26BD6DUPWYJJGLQXB46YJ5CARHMUBLA, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.258712494 (249h0m36.852823744s) [noise=true , address=145.79.227.41:4118 , piece_id=Q3UQCGMLKJOFW3CDKHJ2E2ICUM6KBL2NKX4M5PCENUQ3K5UTDGNA, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.258890608 (249h0m36.853001868s) [noise=true , address=145.79.227.43:4170 , piece_id=JW6SHFJPMXIZ7L3JYYDO6KUKTUBXVQQDBWBGNPKIMFDPZ4O5QI6Q, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.259231278 (249h0m36.853342524s) [noise=true , address=193.149.164.143:4697 , piece_id=HBOR2CM3X7YJYHKBFIU7Z5XQSJ4AB5KYA7TRI5BS5PILF4HR6VGQ, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.259558107 (249h0m36.853669354s) [noise=true , address=167.17.72.70:4763 , piece_id=75DFWEDWWNGCEKZUS7N64NGYHQNIE4KVPG3EVPT2N24RYP5MILKQ, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.261570781 (249h0m36.85568203s) [noise=true , address=64.49.13.169:4706 , piece_id=ORWZHFTP3RW6CK3YHQXTMYQE6LZM7ALELP6KGKY2RHQGCAESIOXQ, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.262498978 (249h0m36.856610234s) [noise=true , address=145.79.227.32:4854 , piece_id=QW7BC6FX3QULWLCAAQKD674P57ICEZ2WMCZKBFQKB2CWFNOXTCQA, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.263300818 (249h0m36.857412064s) [noise=true , address=167.17.72.71:4811 , piece_id=BHYIBF4EWX63Z3DVUMJSVPIMYZIKBB2TGEPSEI67MZ4TNILGZOTA, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.264144311 (249h0m36.858255571s) [noise=true , address=145.79.227.49:4360 , piece_id=NM5WRRSYHWCVVWNTFCVXTC33JL67XY6W2GDVKOBMECQUY7M4R6AQ, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.267053375 (249h0m36.861164631s) [noise=true , address=145.79.227.38:4012 , piece_id=4V5SVPHWZTYC4J55Q6RIKH6KSUWRZL5WTQPSELYHEHXWS3FJ46BQ, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.268474497 (249h0m36.862585754s) [noise=true , address=193.149.164.71:4299 , piece_id=4TDPUI6IHC4AW4EAYWVS2EMYG5GMWPXIS2CHMWXQFVZ3BD7ZWWJA, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.269306913 (249h0m36.863418399s) [noise=true , address=77.90.31.16:4764 , piece_id=AAFUBQIOAPB7BA4AL3IMPJWITNSLTWEKEHAGLH6Z74ZZOZQKEQCA, seg_pos={0 0}, upload=239393]: Uploading piece... Apr 18 01:17:26.359061250 (249h0m36.953172494s) [noise=true , address=145.79.227.49:4360 , piece_id=NM5WRRSYHWCVVWNTFCVXTC33JL67XY6W2GDVKOBMECQUY7M4R6AQ, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: rpc: dial tcp 145.79.227.49:4360: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:17:26.417057335 (249h0m37.011168581s) [noise=true , address=167.17.72.70:4763 , piece_id=75DFWEDWWNGCEKZUS7N64NGYHQNIE4KVPG3EVPT2N24RYP5MILKQ, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:12WTok5fX7a9w8kE4dbKceZLDbfDbuZgeeYrT3FTza8rgRJqwHw, address:167.17.72.70:4763): protocol: write tcp 139.59.144.22:60096->167.17.72.70:4763: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:17:26.826952435 (249h0m37.421063677s) [upload=239393]: Next returned nil segment Apr 18 01:17:26.826969155 (249h0m37.421080396s) [upload=239393]: Waiting for error group managing segments... Apr 18 01:17:26.936465540 (249h0m37.530576782s) [noise=true , address=64.49.13.169:4706 , piece_id=ORWZHFTP3RW6CK3YHQXTMYQE6LZM7ALELP6KGKY2RHQGCAESIOXQ, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000 Apr 18 01:17:26.977833399 (249h0m37.571944631s) [noise=true , address=45.91.116.10:4115 , piece_id=MSFHMO2JD5BOQXBGG3OWHCHLRFABDWIIPEXSB7YPZ3VRGBVJQYIQ, seg_pos={0 0}, upload=239393]: Done uploading piece. err: <nil> Apr 18 01:17:27.095947147 (249h0m37.690058388s) [noise=true , address=145.79.227.32:4854 , piece_id=QW7BC6FX3QULWLCAAQKD674P57ICEZ2WMCZKBFQKB2CWFNOXTCQA, seg_pos={0 0}, upload=239393]: Done uploading piece. err: <nil> Apr 18 01:17:27.357286644 (249h0m37.95139789s) [noise=true , address=145.79.227.41:4118 , piece_id=Q3UQCGMLKJOFW3CDKHJ2E2ICUM6KBL2NKX4M5PCENUQ3K5UTDGNA, seg_pos={0 0}, upload=239393]: Done uploading piece. err: <nil> Apr 18 01:17:27.521463851 (249h0m38.115575099s) [noise=true , address=193.149.164.71:4299 , piece_id=4TDPUI6IHC4AW4EAYWVS2EMYG5GMWPXIS2CHMWXQFVZ3BD7ZWWJA, seg_pos={0 0}, upload=239393]: Done uploading piece. err: <nil> Apr 18 01:17:27.546167095 (249h0m38.140278343s) [noise=true , address=193.149.164.66:4963 , piece_id=64GZH3O7QOXZSKTMBAWDQS5D4ZCVHXNVZWZZMSJMPPL4MGED545Q, seg_pos={0 0}, upload=239393]: Done uploading piece. err: <nil> Apr 18 01:17:27.597486752 (249h0m38.191598002s) [noise=true , address=145.79.227.21:4481 , piece_id=NF2TUTM6NPV747EU4NJY7HUKCUBAFJA3TR55J6P256OBCQ2WDOVQ, seg_pos={0 0}, upload=239393]: Done uploading piece. err: <nil> Apr 18 01:17:29.421258948 (249h0m40.015370176s) [noise=true , address=77.90.31.16:4764 , piece_id=AAFUBQIOAPB7BA4AL3IMPJWITNSLTWEKEHAGLH6Z74ZZOZQKEQCA, seg_pos={0 0}, upload=239393]: Done uploading piece. err: <nil> Apr 18 01:17:30.609678408 (249h0m41.203789655s) [noise=true , address=145.79.227.43:4170 , piece_id=JW6SHFJPMXIZ7L3JYYDO6KUKTUBXVQQDBWBGNPKIMFDPZ4O5QI6Q, seg_pos={0 0}, upload=239393]: Done uploading piece. err: <nil> Apr 18 01:17:35.439257335 (249h0m46.03336858s) [noise=true , address=193.149.164.143:4697 , piece_id=HBOR2CM3X7YJYHKBFIU7Z5XQSJ4AB5KYA7TRI5BS5PILF4HR6VGQ, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, address:193.149.164.143:4697): protocol: storage node overloaded, request limit: 2000 Apr 18 01:18:25.861170061 (249h1m36.455281312s) [noise=true , address=145.79.227.33:4899 , piece_id=5SJPB2MDXLGBFS3JEPJCFD2L5MEKITTX5LZVBXFKAFL4W2OAKIJA, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:12b4pZVQRnjcQNn45VTgdsz5fpjYPMAzSXGR5KJ5BP9XHMkRnRw, address:145.79.227.33:4899): protocol: rpc: dial tcp 145.79.227.33:4899: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:18:25.861572472 (249h1m36.455683724s) [noise=true , address=145.79.227.38:4012 , piece_id=4V5SVPHWZTYC4J55Q6RIKH6KSUWRZL5WTQPSELYHEHXWS3FJ46BQ, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:12ZS5nRRvwoF6T7AHt5Z83Z4r1eyzjMQo5yt9keVUa1UE3DR1W2, address:145.79.227.38:4012): protocol: rpc: dial tcp 145.79.227.38:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:18:25.861900243 (249h1m36.456011497s) [noise=true , address=167.17.72.71:4811 , piece_id=BHYIBF4EWX63Z3DVUMJSVPIMYZIKBB2TGEPSEI67MZ4TNILGZOTA, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9, address:167.17.72.71:4811): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:18:25.862312041 (249h1m36.456423295s) [noise=true , address=167.17.72.68:4672 , piece_id=EGYVPZ46GNAPKJHSFSBXRGW75SJUMLOS3BVNO33KVZJSUS564TVQ, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:12dtrjCvkcpwh4oTDjKe7Ate9JtzUw4BKrqpawATirSXUXkSp8r, address:167.17.72.68:4672): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:18:25.862430921 (249h1m36.456542174s) [noise=true , address=145.79.227.11:4754 , piece_id=GSMWQWGDHUQYMOKMKKVLQTNHHQ6XXCN7TNQTG7V5KBKQQNC5LC3Q, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:125aPwoHeFabk6zZ6bkGNj3USJYgTQG3RB1QhhXssj7Vik2AuCs, address:145.79.227.11:4754): protocol: rpc: dial tcp 145.79.227.11:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:18:25.862748417 (249h1m36.456859669s) [noise=true , address=145.79.227.44:4203 , piece_id=FVMY23ZG6QQKMKIIS7BEL26BD6DUPWYJJGLQXB46YJ5CARHMUBLA, seg_pos={0 0}, upload=239393]: Done uploading piece. err: ecclient: upload failed (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU, address:145.79.227.44:4203): protocol: rpc: dial tcp 145.79.227.44:4203: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:18:25.862897065 (249h1m36.457008306s) [seg_pos={0 0}, upload=239393]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Apr 18 01:18:25.966498351 (249h1m36.560609595s) [upload=239393]: Done uploading segments... Apr 18 01:18:25.966543763 (249h1m36.560654992s) [upload=239393]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.108927743s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXXXXXXXXXXX] duration: 5.001763982s trace: https://jaeger.dev.storj.io/trace/2e0d731d8c8ede95 FAIL (2026-04-18 01:18:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001790954s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 50 out of 60 is failed (threshold=3) [.........XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.109187106s trace: https://jaeger.dev.storj.io/trace/1f0bf0fa2a38a62e FAIL (2026-04-18 01:17:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== _pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.993316623 (248h59m36.587427874s) [noise=true , address=145.79.227.21:4508 , piece_id=7PU7JSOHO5NOXJCRI52ZRT7WZ7A5CDWEHNQITEYDPIHTLB4QT7LQ, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.993814908 (248h59m36.587926149s) [noise=true , address=193.149.164.66:4983 , piece_id=TYO6N3KY22NX2BXET4BBENLOLXMGAHEHFVL2MAR433YKCML4ROKQ, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.994097465 (248h59m36.588208715s) [noise=true , address=145.79.227.43:4171 , piece_id=ESNIO2QAXEQMSWIODBDXCVBDHI3MG2JTAL3Z6U7J4ACXZ55PXGXA, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.994259992 (248h59m36.588371243s) [noise=true , address=145.79.227.32:4848 , piece_id=ZOC5SHWNM32NNAUUU33BTLMKH5HJARUA44G5NGTXEHYQ5JJBRZUQ, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.994652288 (248h59m36.588763539s) [noise=true , address=145.79.227.44:4228 , piece_id=Q2GR2CP4LSEEKBSBHEVPRNU2Z5ZAS7E6ALLWHQVNWENQKAJDEPJA, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.994661120 (248h59m36.588772364s) [noise=true , address=45.91.116.14:4201 , piece_id=YHTEDVYA25FRVPUJ3TB2MOP4EDNOPJY4G3DK6RNQBHOICUYCV3FA, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.995113138 (248h59m36.589224388s) [noise=true , address=193.149.164.143:4696 , piece_id=RDEB4YNY5KAOBRU7ET6O7PARFBPS2CJYUCOJTVKTHWEZEE2ZPLPA, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.995644142 (248h59m36.589755393s) [noise=true , address=77.90.31.12:4617 , piece_id=IB27M3UDB5U7CZS3JDKS56ODA4MTBNYW24RMRBCZVGS7F4I2NNRA, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.996116988 (248h59m36.590228241s) [noise=true , address=193.149.164.71:4302 , piece_id=7ZUBQUOT3CAWOLGFLB474KJUDPUYEMMV2E22Q3I3K4LLJTRYQKOA, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.996742893 (248h59m36.590854144s) [noise=true , address=167.17.72.69:4712 , piece_id=YO3WVRFVPCUE7AJ53DKDGYRG6BJQ3EW5S2TSZOQ6JSSFRRWHPSNA, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.997175154 (248h59m36.591286408s) [noise=true , address=167.17.72.68:4665 , piece_id=YKPE44SVW5MUNU4SMIKDXNQXTA6V7UEMOGRTXV3JAQB7GESOI2UQ, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.997601945 (248h59m36.591713195s) [noise=true , address=77.90.31.17:4787 , piece_id=CBHIXCWQ7OUXFFKH46DTXKK3DNDP2JAKQAOZ4JDHBWQPEJU3V6QQ, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:25.998744476 (248h59m36.592855723s) [noise=true , address=145.79.227.34:4938 , piece_id=SGQII4V6EYSIXDKFNAAE2L4FOGUCGRNSNC5OJ4NRTJ3XLLICANCA, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:26.000014353 (248h59m36.594125608s) [noise=true , address=167.17.72.71:4809 , piece_id=V7RHFCHSGINFTE23KI5GZP3GIDDBX2KVXQ7YIFF7JS6N5TBTGG2Q, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:26.006454469 (248h59m36.600565716s) [noise=true , address=145.79.227.38:4008 , piece_id=D5EDZFGU7HVSU2MIBNDLDWFZ3XXLJNPBUTDOTU24WENYJLHNNW4A, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:26.010378125 (248h59m36.604489365s) [noise=true , address=145.79.227.11:4745 , piece_id=RK6YU7KECKJFIO7BRQ7A6H5DFBIQJ3WJ45QJA2ZWS6CBQBM42NKQ, seg_pos={0 0}, upload=239377]: Uploading piece... Apr 18 01:16:26.145063351 (248h59m36.739174602s) [noise=true , address=167.17.72.68:4665 , piece_id=YKPE44SVW5MUNU4SMIKDXNQXTA6V7UEMOGRTXV3JAQB7GESOI2UQ, seg_pos={0 0}, upload=239377]: Done uploading piece. err: ecclient: upload failed (node:1ZizL8yPSYhNwNHYRhmTsY8g8MYCibE3gusVRgyU6DUKgt3jkr, address:167.17.72.68:4665): protocol: write tcp 139.59.144.22:40480->167.17.72.68:4665: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:16:26.558820084 (248h59m37.152931324s) [upload=239377]: Next returned nil segment Apr 18 01:16:26.558838413 (248h59m37.152949662s) [upload=239377]: Waiting for error group managing segments... Apr 18 01:16:26.708109911 (248h59m37.302221159s) [noise=true , address=145.79.227.32:4848 , piece_id=ZOC5SHWNM32NNAUUU33BTLMKH5HJARUA44G5NGTXEHYQ5JJBRZUQ, seg_pos={0 0}, upload=239377]: Done uploading piece. err: <nil> Apr 18 01:16:27.263991723 (248h59m37.858102972s) [noise=true , address=193.149.164.71:4302 , piece_id=7ZUBQUOT3CAWOLGFLB474KJUDPUYEMMV2E22Q3I3K4LLJTRYQKOA, seg_pos={0 0}, upload=239377]: Done uploading piece. err: <nil> Apr 18 01:16:27.277799911 (248h59m37.871911157s) [noise=true , address=193.149.164.66:4983 , piece_id=TYO6N3KY22NX2BXET4BBENLOLXMGAHEHFVL2MAR433YKCML4ROKQ, seg_pos={0 0}, upload=239377]: Done uploading piece. err: <nil> Apr 18 01:16:27.880271740 (248h59m38.474382979s) [noise=true , address=64.49.13.140:4513 , piece_id=UZ2MOEAUSUZD2GXGBIRV6OIESROVEFUXGDXHGXMFUIRMXRF363FA, seg_pos={0 0}, upload=239377]: Done uploading piece. err: <nil> Apr 18 01:16:28.646697435 (248h59m39.240808681s) [noise=true , address=45.91.116.14:4201 , piece_id=YHTEDVYA25FRVPUJ3TB2MOP4EDNOPJY4G3DK6RNQBHOICUYCV3FA, seg_pos={0 0}, upload=239377]: Done uploading piece. err: ecclient: upload failed (node:1RUqtJWMppPz7u7bTYm5chvDf53HVDPCwC7bnmTs1R5hJD8Chp, address:45.91.116.14:4201): protocol: storage node overloaded, request limit: 2000 Apr 18 01:16:29.002735700 (248h59m39.596846948s) [noise=true , address=77.90.31.12:4617 , piece_id=IB27M3UDB5U7CZS3JDKS56ODA4MTBNYW24RMRBCZVGS7F4I2NNRA, seg_pos={0 0}, upload=239377]: Done uploading piece. err: <nil> Apr 18 01:16:31.528893245 (248h59m42.123004484s) [noise=true , address=145.79.227.21:4508 , piece_id=7PU7JSOHO5NOXJCRI52ZRT7WZ7A5CDWEHNQITEYDPIHTLB4QT7LQ, seg_pos={0 0}, upload=239377]: Done uploading piece. err: <nil> Apr 18 01:16:33.185749935 (248h59m43.779861192s) [noise=true , address=77.90.31.11:4573 , piece_id=64UXG5VDUPA7C2ACRNTEOQZUX3I5UWDTN6KCD6BLNKQ3UKZWJXWQ, seg_pos={0 0}, upload=239377]: Done uploading piece. err: <nil> Apr 18 01:16:35.826445770 (248h59m46.420557013s) [noise=true , address=77.90.31.17:4787 , piece_id=CBHIXCWQ7OUXFFKH46DTXKK3DNDP2JAKQAOZ4JDHBWQPEJU3V6QQ, seg_pos={0 0}, upload=239377]: Done uploading piece. err: <nil> Apr 18 01:16:52.550946454 (249h0m3.145057694s) [noise=true , address=145.79.227.43:4171 , piece_id=ESNIO2QAXEQMSWIODBDXCVBDHI3MG2JTAL3Z6U7J4ACXZ55PXGXA, seg_pos={0 0}, upload=239377]: Done uploading piece. err: <nil> Apr 18 01:17:25.587129023 (249h0m36.181240274s) [noise=true , address=145.79.227.44:4228 , piece_id=Q2GR2CP4LSEEKBSBHEVPRNU2Z5ZAS7E6ALLWHQVNWENQKAJDEPJA, seg_pos={0 0}, upload=239377]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:145.79.227.44:4228): protocol: rpc: dial tcp 145.79.227.44:4228: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:17:25.587483809 (249h0m36.181595062s) [noise=true , address=145.79.227.34:4938 , piece_id=SGQII4V6EYSIXDKFNAAE2L4FOGUCGRNSNC5OJ4NRTJ3XLLICANCA, seg_pos={0 0}, upload=239377]: Done uploading piece. err: ecclient: upload failed (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim, address:145.79.227.34:4938): protocol: rpc: dial tcp 145.79.227.34:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:17:25.588533255 (249h0m36.182644518s) [noise=true , address=167.17.72.69:4712 , piece_id=YO3WVRFVPCUE7AJ53DKDGYRG6BJQ3EW5S2TSZOQ6JSSFRRWHPSNA, seg_pos={0 0}, upload=239377]: Done uploading piece. err: ecclient: upload failed (node:12XvwSGGHq6iMFzZoTwCvhoQDupaaZ76sLimGXZWqF7UyVM6QqY, address:167.17.72.69:4712): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:17:25.588633200 (249h0m36.182744452s) [noise=true , address=167.17.72.71:4809 , piece_id=V7RHFCHSGINFTE23KI5GZP3GIDDBX2KVXQ7YIFF7JS6N5TBTGG2Q, seg_pos={0 0}, upload=239377]: Done uploading piece. err: ecclient: upload failed (node:1pqQxA4MZEtsh9u15s9vCDpfxGN96Fx58SPHFhGiaW6hysMsFC, address:167.17.72.71:4809): protocol: write tcp 139.59.144.22:56174->167.17.72.71:4809: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:17:25.588798020 (249h0m36.182909271s) [noise=true , address=193.149.164.143:4696 , piece_id=RDEB4YNY5KAOBRU7ET6O7PARFBPS2CJYUCOJTVKTHWEZEE2ZPLPA, seg_pos={0 0}, upload=239377]: Done uploading piece. err: ecclient: upload failed (node:1p6mcvX3TBdnYy5Eaugk1t5KPfoeJT1sxcg5TX8aSRGVhHGSsy, address:193.149.164.143:4696): protocol: read tcp 139.59.144.22:56154->193.149.164.143:4696: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:17:25.588910601 (249h0m36.183021851s) [noise=true , address=145.79.227.11:4745 , piece_id=RK6YU7KECKJFIO7BRQ7A6H5DFBIQJ3WJ45QJA2ZWS6CBQBM42NKQ, seg_pos={0 0}, upload=239377]: Done uploading piece. err: ecclient: upload failed (node:1H9sLaFPDPZxKvwXho76wrqPiFe9yu87Ue9BT4ip92qfLmjbnF, address:145.79.227.11:4745): protocol: read tcp 139.59.144.22:53762->145.79.227.11:4745: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:17:25.589186872 (249h0m36.183298123s) [noise=true , address=145.79.227.38:4008 , piece_id=D5EDZFGU7HVSU2MIBNDLDWFZ3XXLJNPBUTDOTU24WENYJLHNNW4A, seg_pos={0 0}, upload=239377]: Done uploading piece. err: ecclient: upload failed (node:12KYDzfd4KtZm1cyRXCKyntNLnoLYCBdPubsebpdi8H6khSpeAN, address:145.79.227.38:4008): protocol: read tcp 139.59.144.22:34998->145.79.227.38:4008: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:17:25.589252085 (249h0m36.183363333s) [seg_pos={0 0}, upload=239377]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Apr 18 01:17:25.693343367 (249h0m36.287454619s) [upload=239377]: Done uploading segments... Apr 18 01:17:25.693405228 (249h0m36.287516483s) [upload=239377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.109218339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [X..X..X....X....X.....X....XX..X..........XX...XX.XXX.XXXX..] duration: 5.92751946s trace: https://jaeger.dev.storj.io/trace/305953df7fc505d3 FAIL (2026-04-18 01:17:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.927646332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXXXXXXXXXX] duration: 5.000704605s trace: https://jaeger.dev.storj.io/trace/2669acb53c524a92 FAIL (2026-04-18 01:17:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000745835s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [.X..X..X....X....X.....X....XX..X..........XX...XX.XXX.XXXX.] duration: 6.633157393s trace: https://jaeger.dev.storj.io/trace/305953df7fc505d3 FAIL (2026-04-18 01:16:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.633175791s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [..X..X..X....X....X.....X....XX..X..........XX...XX.XXX.XXXX] duration: 1m0.000838075s trace: https://jaeger.dev.storj.io/trace/305953df7fc505d3 FAIL (2026-04-18 01:16:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000851558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 49 out of 60 is failed (threshold=3) [..........XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.110469745s trace: https://jaeger.dev.storj.io/trace/30d94db096841faa FAIL (2026-04-18 01:16:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== r 18 01:15:25.735284566 (248h58m36.329395811s) [noise=true , address=145.79.227.49:4368 , piece_id=SDDWHKWSP3NVUPHVJN7MD3INKJAYUQ3QP5WK224Y6MBVALBQFOEA, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.735758128 (248h58m36.329869373s) [noise=true , address=45.91.116.5:4316 , piece_id=PVB2HNFVO2Y5YPGJ4XF575NE3CRZDW64GOQIHOAPSOXKVR7CM3NQ, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.736139143 (248h58m36.330250388s) [noise=true , address=145.79.227.11:4759 , piece_id=3V3TYUQCT74GFIQCKKR3SXF66LVETYCV7XPX3XEPKRBRX52NSDFA, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.736625468 (248h58m36.33073671s) [noise=true , address=167.17.72.70:4772 , piece_id=VQVQI4ZGRN4RCBHGNUODEAMX7EPDVONGW2S5YVLGDNGOVYOTOBSQ, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.737023101 (248h58m36.331134347s) [noise=true , address=145.79.227.21:4508 , piece_id=QTRT4QS5VAOSMUIXFYAQZYJTBB225SBPQWWWLWYPSRYLSVAKWDGQ, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.737373181 (248h58m36.331484423s) [noise=true , address=145.79.227.34:4938 , piece_id=RRFBRU3OOLOOAA2TNWNL7DJZV7NJPWYKMKKKD76E3Z4IJRG3DQTA, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.737689717 (248h58m36.331800959s) [noise=true , address=145.79.227.28:4738 , piece_id=PTUD74V6CN73BYR2OC7FELQIHS4ZLO3DBZTGEFFG5TPZULEPUKTA, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.738051020 (248h58m36.332162252s) [noise=true , address=145.79.227.29:4966 , piece_id=I54FANLT3R33LMVMVHKH6MBMERYMRWF3VVAPBPE6RNM4EKVNSMSA, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.738477277 (248h58m36.332588523s) [noise=true , address=193.149.164.66:4988 , piece_id=SMOQ735PWTP43I2AAG2DTAE62S2VQFND7NOJBUFYLPHIJFRIFY7Q, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.738936723 (248h58m36.333047967s) [noise=true , address=77.90.31.19:4860 , piece_id=DB4MKRPCP4D5JW73TVCFM7OJIVPWWR2UOHDLWWGHVXYFZHTNNEPA, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.739356036 (248h58m36.333467279s) [noise=true , address=145.79.227.32:4852 , piece_id=VMMADGBCTYN4T3BOBBGETWBBWC7UNJ6VKGNE4GPWQWLEV7VFBZ3A, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.739420502 (248h58m36.333531742s) [noise=true , address=45.91.116.4:4971 , piece_id=QL3IZRIDDMJE2U2NDWAF4TBI5LZFNTLYWNZZSD2AWDEWTW3ILNBA, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.739820947 (248h58m36.333932194s) [noise=true , address=167.17.72.68:4658 , piece_id=EM5Z4F4EUJ2GNLXMJZHQXPNYJX6EPCARN5UVDPQYO2TMPGGLQLLQ, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.740317532 (248h58m36.334428781s) [noise=true , address=167.17.72.71:4811 , piece_id=FPOPHSP737T3UWJFOPUHTXP7ALI2DIAMHRM576OOPHPA5456JLSA, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:25.740841397 (248h58m36.334952644s) [noise=true , address=77.90.31.17:4781 , piece_id=WZECJBOLVNMFIZIIDHSR2ERTM6KFK7I4E26BVV7BKN4VYZL4S7QA, seg_pos={0 0}, upload=239358]: Uploading piece... Apr 18 01:15:26.175071171 (248h58m36.769182421s) [noise=true , address=145.79.227.21:4508 , piece_id=QTRT4QS5VAOSMUIXFYAQZYJTBB225SBPQWWWLWYPSRYLSVAKWDGQ, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:145.79.227.21:4508): protocol: storage node overloaded, request limit: 2000 Apr 18 01:15:26.440455484 (248h58m37.034566737s) [noise=true , address=77.90.31.19:4860 , piece_id=DB4MKRPCP4D5JW73TVCFM7OJIVPWWR2UOHDLWWGHVXYFZHTNNEPA, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:17ddUMj4ZejcjjWGzkGEM63vbCoEPnu7ePznzAPyzQofW8Q4gd, address:77.90.31.19:4860): protocol: storage node overloaded, request limit: 2000 Apr 18 01:15:26.501201114 (248h58m37.095312343s) [upload=239358]: Next returned nil segment Apr 18 01:15:26.501218162 (248h58m37.095329411s) [upload=239358]: Waiting for error group managing segments... Apr 18 01:15:26.502458183 (248h58m37.096569438s) [noise=true , address=167.17.72.68:4658 , piece_id=EM5Z4F4EUJ2GNLXMJZHQXPNYJX6EPCARN5UVDPQYO2TMPGGLQLLQ, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:12kZqHgYeRqLKTVYyMPQp7o2XL5ApzAXGorjSPSg41g43cKW8eB, address:167.17.72.68:4658): protocol: storage node overloaded, request limit: 2000 Apr 18 01:15:26.698814630 (248h58m37.292925876s) [noise=true , address=145.79.227.28:4738 , piece_id=PTUD74V6CN73BYR2OC7FELQIHS4ZLO3DBZTGEFFG5TPZULEPUKTA, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:1E3hj2265uPVRxbFiXEbkQsuDrfMe7HbRzxoYLfHNVm4vQJMQd, address:145.79.227.28:4738): protocol: storage node overloaded, request limit: 2000 Apr 18 01:15:27.023491727 (248h58m37.617602973s) [noise=true , address=193.149.164.66:4988 , piece_id=SMOQ735PWTP43I2AAG2DTAE62S2VQFND7NOJBUFYLPHIJFRIFY7Q, seg_pos={0 0}, upload=239358]: Done uploading piece. err: <nil> Apr 18 01:15:27.030799154 (248h58m37.624910427s) [noise=true , address=193.149.164.71:4317 , piece_id=VVIKU3NR326N3ICL2SS423NXQTW3C2VSSWLTUL6K3IPNLEXPR7XQ, seg_pos={0 0}, upload=239358]: Done uploading piece. err: <nil> Apr 18 01:15:29.216413511 (248h58m39.810524757s) [noise=true , address=45.91.116.4:4971 , piece_id=QL3IZRIDDMJE2U2NDWAF4TBI5LZFNTLYWNZZSD2AWDEWTW3ILNBA, seg_pos={0 0}, upload=239358]: Done uploading piece. err: <nil> Apr 18 01:15:29.713202647 (248h58m40.307313887s) [noise=true , address=64.49.13.140:4513 , piece_id=YL4V2YUK5RLI2D44HWN4EYRVBBEQJXRBA7EXM77IFZPINJMMYJGQ, seg_pos={0 0}, upload=239358]: Done uploading piece. err: <nil> Apr 18 01:15:31.544281134 (248h58m42.138392373s) [noise=true , address=45.91.116.5:4316 , piece_id=PVB2HNFVO2Y5YPGJ4XF575NE3CRZDW64GOQIHOAPSOXKVR7CM3NQ, seg_pos={0 0}, upload=239358]: Done uploading piece. err: <nil> Apr 18 01:15:35.115194197 (248h58m45.709305443s) [noise=true , address=77.90.31.17:4781 , piece_id=WZECJBOLVNMFIZIIDHSR2ERTM6KFK7I4E26BVV7BKN4VYZL4S7QA, seg_pos={0 0}, upload=239358]: Done uploading piece. err: <nil> Apr 18 01:16:25.319378267 (248h59m35.913489518s) [noise=true , address=145.79.227.32:4852 , piece_id=VMMADGBCTYN4T3BOBBGETWBBWC7UNJ6VKGNE4GPWQWLEV7VFBZ3A, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:1CWXaz1RJdY28Y4jCshs5Ti4GhJKR1LJL1pBBEWGWrzbfZXfav, address:145.79.227.32:4852): protocol: rpc: dial tcp 145.79.227.32:4852: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:16:25.319705689 (248h59m35.913816944s) [noise=true , address=167.17.72.70:4772 , piece_id=VQVQI4ZGRN4RCBHGNUODEAMX7EPDVONGW2S5YVLGDNGOVYOTOBSQ, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:12PGtQc63J4cgYv18pkTeE639aFrf6mMp24xkWjynWnibe4QHFH, address:167.17.72.70:4772): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:16:25.320090269 (248h59m35.914201546s) [noise=true , address=193.149.164.143:4703 , piece_id=G4DGFWAMQG5ZR7VQE4DRPMEMLS7HRLBQTPOSQHBRETFOIXT474LQ, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:12eomwcuHK58QuxBp3sAARJjm6Doc7zEzRYCodsTzFT4v6Nti9g, address:193.149.164.143:4703): protocol: rpc: dial tcp 193.149.164.143:4703: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:16:25.320593323 (248h59m35.914704569s) [noise=true , address=167.17.72.71:4811 , piece_id=FPOPHSP737T3UWJFOPUHTXP7ALI2DIAMHRM576OOPHPA5456JLSA, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9, address:167.17.72.71:4811): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:16:25.320696769 (248h59m35.914808017s) [noise=true , address=145.79.227.29:4966 , piece_id=I54FANLT3R33LMVMVHKH6MBMERYMRWF3VVAPBPE6RNM4EKVNSMSA, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:1DLnsYVdudE4mY9dNAQKhBRppUEAHvxbf84KX69451pwxSxzZK, address:145.79.227.29:4966): protocol: rpc: dial tcp 145.79.227.29:4966: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:16:25.320875582 (248h59m35.914986836s) [noise=true , address=145.79.227.11:4759 , piece_id=3V3TYUQCT74GFIQCKKR3SXF66LVETYCV7XPX3XEPKRBRX52NSDFA, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:12RFJ2jqXCSsXgc8ZDNTptWcaUV5Lm25Eobt58t7vGiQY34uhXz, address:145.79.227.11:4759): protocol: rpc: dial tcp 145.79.227.11:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:16:25.320950057 (248h59m35.915061307s) [noise=true , address=145.79.227.49:4368 , piece_id=SDDWHKWSP3NVUPHVJN7MD3INKJAYUQ3QP5WK224Y6MBVALBQFOEA, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:12Q6suyRuFmW1rG5qq8faCFbCkfAPaNfYQaBv6qhL1DNavUffkB, address:145.79.227.49:4368): protocol: rpc: dial tcp 145.79.227.49:4368: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:16:25.321025607 (248h59m35.915136854s) [noise=true , address=145.79.227.34:4938 , piece_id=RRFBRU3OOLOOAA2TNWNL7DJZV7NJPWYKMKKKD76E3Z4IJRG3DQTA, seg_pos={0 0}, upload=239358]: Done uploading piece. err: ecclient: upload failed (node:128NJghaGo1dYxjSyjWksexXS89mgdtPH4GaTrwSavrqNfWabim, address:145.79.227.34:4938): protocol: rpc: dial tcp 145.79.227.34:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:16:25.321084537 (248h59m35.91519578s) [seg_pos={0 0}, upload=239358]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Apr 18 01:16:25.426364077 (248h59m36.020475317s) [upload=239358]: Done uploading segments... Apr 18 01:16:25.426407915 (248h59m36.020519162s) [upload=239358]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.110491531s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXXXXXXXXX] duration: 5.001111477s trace: https://jaeger.dev.storj.io/trace/75d34739b67856d2 FAIL (2026-04-18 01:15:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001124983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [X..X..X..X....X....X.....X....XX..X..........XX...XX.XXX.XXX] duration: 1m0.096233374s trace: https://jaeger.dev.storj.io/trace/36bc1325f912b332 FAIL (2026-04-18 01:15:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.096271046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 48 out of 60 is failed (threshold=3) [...........XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.108958463s trace: https://jaeger.dev.storj.io/trace/2503993d20c3a50b FAIL (2026-04-18 01:15:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 90.31.19:4860 , piece_id=MGYNPA7NJZJBCJYDPZ7GCMQJVJCFWA6OBJIGKFTPEC57G5LXFDTA, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.463160606 (248h57m36.057271858s) [noise=true , address=77.90.31.18:4814 , piece_id=XAQASPVVVWJOLYTREKF2QVQTQWNNC2C6NQLIM3HSAEDAPYLHP64Q, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.463177583 (248h57m36.057288828s) [noise=true , address=145.79.227.13:4816 , piece_id=NY6QWA2PREIISFZ36EESVLUN5NMG2XEAUVG4HUXPYNLQQZ2DQNBQ, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.463918221 (248h57m36.058029471s) [noise=true , address=145.79.227.11:4755 , piece_id=QET5OBMQMSTFS344WHES43E7FVHVMZXTBNIU5EJSE6OWWCQCUFPQ, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.463927869 (248h57m36.058039124s) [noise=true , address=193.149.164.71:4319 , piece_id=RMCPEX3MUB4VQKYO5EXR4YXNJ2GMZJR3GTHGO7RNKWPVLHN3LIWQ, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.464642099 (248h57m36.058753352s) [noise=true , address=145.79.227.44:4211 , piece_id=N4PGDF5BU2ON3I4SOZA4RXY7MBTXYQ7C6DGE7NLYUN3DIQ7NMZDA, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.464877227 (248h57m36.058988477s) [noise=true , address=167.17.72.68:4639 , piece_id=OAMFYCJWSANSHJ5PVYCGHTLY7O7UJYVDIIRJEY6BQQWD5RVPIIYA, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.465253095 (248h57m36.059364349s) [noise=true , address=145.79.227.32:4840 , piece_id=64C5HMMDS7IVNFWJJIMYZXCS7IUJ5IDC6NFN36X6UUNLIOIMGRHA, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.465403936 (248h57m36.059515183s) [noise=true , address=167.17.72.71:4781 , piece_id=HTY4EG6QKFAIKHJG2B46OOIM3AUUC4CHMPGC4W5LCXHQ6HYPXW5A, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.465908621 (248h57m36.060019887s) [noise=true , address=145.79.227.33:4875 , piece_id=HW2NLFFHKGVSWUCP56NIJUDZYDIGP7TDNKIEDSMAOO7A6XKFJLEQ, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.465973565 (248h57m36.060084806s) [noise=true , address=77.90.31.11:4573 , piece_id=OFZY7RX4XFEQ65KZ3QS76J3FD6S7JALJ2CM2FCUVL7DBTMH3AOEA, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.466593293 (248h57m36.060704544s) [noise=true , address=64.49.13.138:4488 , piece_id=NQBMLEWJ7I6N4HUUSJXGGTJBI547RAQI36ZLAC2VULR55QCL3MIA, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.466843978 (248h57m36.060955228s) [noise=true , address=167.17.72.70:4772 , piece_id=HVV55WUJQESFPHUZB7P77F7PZMBJHIIKMT2ZTZ6OAOFXUSXKEP3Q, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.467206078 (248h57m36.061317326s) [noise=true , address=77.90.31.17:4781 , piece_id=E7IHQTKWRV7J5FSPTER6BCJTOZVRVA7NDUPEXZX7ZZRJCYEZVL6Q, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.467750617 (248h57m36.061861871s) [noise=true , address=193.149.164.143:4704 , piece_id=KESEQXFD7IDEZVX4EPRGZUKHMPSQ27V45JEX453OXJFWPVGGVRQQ, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.470920244 (248h57m36.065031472s) [noise=true , address=167.17.72.69:4735 , piece_id=ZRWNIIIDP6RG32RK5YE7RCF2G5DLCZEDIIB4CW7K6LO3H73IWH4Q, seg_pos={0 0}, upload=239344]: Uploading piece... Apr 18 01:14:25.640671396 (248h57m36.234782646s) [noise=true , address=167.17.72.71:4781 , piece_id=HTY4EG6QKFAIKHJG2B46OOIM3AUUC4CHMPGC4W5LCXHQ6HYPXW5A, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:122DUAu2vUiUawKG4APwaTUoLrJVgk4tYQDaZchtKfm46sU27hg, address:167.17.72.71:4781): protocol: write tcp 139.59.144.22:55370->167.17.72.71:4781: write: connection refused; write tcp 139.59.144.22:55370->167.17.72.71:4781: write: connection refused Apr 18 01:14:26.046862056 (248h57m36.640973297s) [upload=239344]: Next returned nil segment Apr 18 01:14:26.046880021 (248h57m36.640991272s) [upload=239344]: Waiting for error group managing segments... Apr 18 01:14:26.217856854 (248h57m36.811968099s) [noise=true , address=64.49.13.169:4706 , piece_id=XUB6VIIZVBVVG5M3KKYWEP22ZXSHYSZTUMFZBUCSCR6BKTLCRPQA, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: storage node overloaded, request limit: 2000 Apr 18 01:14:26.231887766 (248h57m36.825999005s) [noise=true , address=145.79.227.32:4840 , piece_id=64C5HMMDS7IVNFWJJIMYZXCS7IUJ5IDC6NFN36X6UUNLIOIMGRHA, seg_pos={0 0}, upload=239344]: Done uploading piece. err: <nil> Apr 18 01:14:26.722888398 (248h57m37.316999638s) [noise=true , address=64.49.13.138:4488 , piece_id=NQBMLEWJ7I6N4HUUSJXGGTJBI547RAQI36ZLAC2VULR55QCL3MIA, seg_pos={0 0}, upload=239344]: Done uploading piece. err: <nil> Apr 18 01:14:26.736258723 (248h57m37.330369965s) [noise=true , address=193.149.164.71:4319 , piece_id=RMCPEX3MUB4VQKYO5EXR4YXNJ2GMZJR3GTHGO7RNKWPVLHN3LIWQ, seg_pos={0 0}, upload=239344]: Done uploading piece. err: <nil> Apr 18 01:14:28.235429053 (248h57m38.829540294s) [noise=true , address=145.79.227.13:4816 , piece_id=NY6QWA2PREIISFZ36EESVLUN5NMG2XEAUVG4HUXPYNLQQZ2DQNBQ, seg_pos={0 0}, upload=239344]: Done uploading piece. err: <nil> Apr 18 01:14:28.655013166 (248h57m39.249124413s) [noise=true , address=77.90.31.11:4573 , piece_id=OFZY7RX4XFEQ65KZ3QS76J3FD6S7JALJ2CM2FCUVL7DBTMH3AOEA, seg_pos={0 0}, upload=239344]: Done uploading piece. err: <nil> Apr 18 01:14:28.845970495 (248h57m39.440081742s) [noise=true , address=77.90.31.19:4860 , piece_id=MGYNPA7NJZJBCJYDPZ7GCMQJVJCFWA6OBJIGKFTPEC57G5LXFDTA, seg_pos={0 0}, upload=239344]: Done uploading piece. err: <nil> Apr 18 01:14:32.906431539 (248h57m43.500542805s) [noise=true , address=77.90.31.18:4814 , piece_id=XAQASPVVVWJOLYTREKF2QVQTQWNNC2C6NQLIM3HSAEDAPYLHP64Q, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: storage node overloaded, request limit: 2000 Apr 18 01:14:35.201499357 (248h57m45.795610597s) [noise=true , address=77.90.31.17:4781 , piece_id=E7IHQTKWRV7J5FSPTER6BCJTOZVRVA7NDUPEXZX7ZZRJCYEZVL6Q, seg_pos={0 0}, upload=239344]: Done uploading piece. err: <nil> Apr 18 01:15:25.055224981 (248h58m35.649336233s) [noise=true , address=167.17.72.69:4735 , piece_id=ZRWNIIIDP6RG32RK5YE7RCF2G5DLCZEDIIB4CW7K6LO3H73IWH4Q, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:124GDKoErXTm1iZavJkmTkcgWAK91AVBaJitTYnUjrD8dh4tawp, address:167.17.72.69:4735): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:15:25.055451089 (248h58m35.64956234s) [noise=true , address=145.79.227.44:4211 , piece_id=N4PGDF5BU2ON3I4SOZA4RXY7MBTXYQ7C6DGE7NLYUN3DIQ7NMZDA, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:145.79.227.44:4211): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:15:25.055636430 (248h58m35.649747683s) [noise=true , address=167.17.72.70:4772 , piece_id=HVV55WUJQESFPHUZB7P77F7PZMBJHIIKMT2ZTZ6OAOFXUSXKEP3Q, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:12PGtQc63J4cgYv18pkTeE639aFrf6mMp24xkWjynWnibe4QHFH, address:167.17.72.70:4772): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:15:25.055972904 (248h58m35.650084195s) [noise=true , address=145.79.227.34:4932 , piece_id=LMFFGUTM57XYYODXSGDOAVN7U7IJ7HK33DU3L43KZNFMNHW3VZLA, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm, address:145.79.227.34:4932): protocol: rpc: dial tcp 145.79.227.34:4932: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:15:25.056353713 (248h58m35.650464967s) [noise=true , address=145.79.227.11:4755 , piece_id=QET5OBMQMSTFS344WHES43E7FVHVMZXTBNIU5EJSE6OWWCQCUFPQ, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:145.79.227.11:4755): protocol: rpc: dial tcp 145.79.227.11:4755: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:15:25.056732628 (248h58m35.650843879s) [noise=true , address=145.79.227.33:4875 , piece_id=HW2NLFFHKGVSWUCP56NIJUDZYDIGP7TDNKIEDSMAOO7A6XKFJLEQ, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: read tcp 139.59.144.22:60506->145.79.227.33:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:15:25.056903162 (248h58m35.651014412s) [noise=true , address=193.149.164.143:4704 , piece_id=KESEQXFD7IDEZVX4EPRGZUKHMPSQ27V45JEX453OXJFWPVGGVRQQ, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY, address:193.149.164.143:4704): protocol: rpc: dial tcp 193.149.164.143:4704: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:15:25.057365743 (248h58m35.651476994s) [noise=true , address=167.17.72.68:4639 , piece_id=OAMFYCJWSANSHJ5PVYCGHTLY7O7UJYVDIIRJEY6BQQWD5RVPIIYA, seg_pos={0 0}, upload=239344]: Done uploading piece. err: ecclient: upload failed (node:1yUZFpmyPoPBFxxM7MrUzngAed4ZnNBAZs9gsD9fVPUVoBtB7, address:167.17.72.68:4639): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:15:25.057400592 (248h58m35.651511819s) [seg_pos={0 0}, upload=239344]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Apr 18 01:15:25.161487864 (248h58m35.755599108s) [upload=239344]: Done uploading segments... Apr 18 01:15:25.161532067 (248h58m35.755643314s) [upload=239344]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.108975558s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXXXXXXXX] duration: 5.001818903s trace: https://jaeger.dev.storj.io/trace/7eedb7ffb9ca48a2 FAIL (2026-04-18 01:15:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00189653s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.X..X..X..X....X....X.....X....XX..X..........XX...XX.XXX.XX] duration: 1m0.001632549s trace: https://jaeger.dev.storj.io/trace/788fefe6d52ed7ed FAIL (2026-04-18 01:14:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001648065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 47 out of 60 is failed (threshold=3) [............XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.106678166s trace: https://jaeger.dev.storj.io/trace/25d2a2e855747955 FAIL (2026-04-18 01:14:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== noise=true , address=145.79.227.32:4856 , piece_id=WS5BAD3VQMLAIK5ZDJPY6A5JIZ6OZMJELUAT22XYTQCVAKOH7MSA, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.125874868 (248h56m35.719986118s) [noise=true , address=145.79.227.43:4167 , piece_id=QDE3WXXUVBI6FHYI26BNUTJF5ZUXKUBWGF3KL6I5GCONWLEXJXKA, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.126456248 (248h56m35.720567498s) [noise=true , address=145.79.227.33:4885 , piece_id=377XYUXSXT6TSSRG442UQRAULJTG7QKHJEJR5VJ6MDLVATU6NI5Q, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.126874363 (248h56m35.720985609s) [noise=true , address=167.17.72.69:4711 , piece_id=OLL6CAEQNA75W6TFB7KZRS2VNGDAM2QPJL2OA3UQIVJXGX4OARDQ, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.127045420 (248h56m35.721156669s) [noise=true , address=167.17.72.68:4666 , piece_id=JUCG5GRM5EG3KRCPTF6RWLDN5XGFV7FJFWUWTRFHNJX7POIAPQXQ, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.127275441 (248h56m35.721386685s) [noise=true , address=64.49.13.138:4483 , piece_id=CP24H3R353N7UHPBDJFBV7SHZW5ZLNHUXA727YSTWLFUUL2YH65Q, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.127628679 (248h56m35.721739922s) [noise=true , address=45.91.116.11:4140 , piece_id=AXB5HLQGR7PXL7YZK25NFRCC6D3FGQQPLS5BFX6ALFYKED5XFOOA, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.127998753 (248h56m35.722109994s) [noise=true , address=167.17.72.71:4789 , piece_id=AOQVFHXSNVWDDF5DSH56R2SFSXW7OOTP3NETVDNECQIFRNHM6ANQ, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.128397729 (248h56m35.722508969s) [noise=true , address=145.79.227.40:4070 , piece_id=C47ABPSAFVDZ2G6PROWB7SAKBKIBIZLWWYLCLK5IJS3TWALDUS4Q, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.128769219 (248h56m35.722880459s) [noise=true , address=193.149.164.66:4986 , piece_id=OQ364K5WVAFPAZALKF2NPXZZRCJZDUX7NAADQXIFRSJXHCVSQD6Q, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.129242083 (248h56m35.723353333s) [noise=true , address=145.79.227.13:4816 , piece_id=FC5OY5KCJ52DAHA7BWUHIFVQFARRN4W3ZQ7MQVF46EQIW7P2BWAA, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.129766107 (248h56m35.723877351s) [noise=true , address=193.149.164.143:4695 , piece_id=I6WAE2PQF75NNSZPPE7GMAS4OKEZNV2GT5YH5Z5GKRTZX7YMEP4A, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.130249808 (248h56m35.724361058s) [noise=true , address=145.79.227.11:4761 , piece_id=WXKL24TUKQRGO6IBAEDH7VVZLZKCHYVNA2BEBSBKRDGH5B3A5HLQ, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.130492769 (248h56m35.72460402s) [noise=true , address=193.149.164.71:4287 , piece_id=URCNFH2WDPJFP5JUFEQPFTUGCJIKJW7YMAZBEQG646J2QU5GVQLA, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.130914284 (248h56m35.725025535s) [noise=true , address=167.17.72.70:4749 , piece_id=NVS3ARCT35NTLUOFK6BBWYWZ6KXP42TP4DAR64TN4IAGVD7FYFKA, seg_pos={0 0}, upload=239326]: Uploading piece... Apr 18 01:13:25.264951532 (248h56m35.859062782s) [noise=true , address=167.17.72.69:4711 , piece_id=OLL6CAEQNA75W6TFB7KZRS2VNGDAM2QPJL2OA3UQIVJXGX4OARDQ, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:12pefcoMB9X8CRc3X8CgJMF9F3uyZkQKDFHirCBM56PiWzLE9hZ, address:167.17.72.69:4711): protocol: write tcp 139.59.144.22:39164->167.17.72.69:4711: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:13:25.270378723 (248h56m35.864489972s) [noise=true , address=167.17.72.68:4666 , piece_id=JUCG5GRM5EG3KRCPTF6RWLDN5XGFV7FJFWUWTRFHNJX7POIAPQXQ, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:121qq3B9v2xUYb5yy1BrCrA1CC3EwZkD3hZH4GJmwV68t9bswC3, address:167.17.72.68:4666): protocol: write tcp 139.59.144.22:40326->167.17.72.68:4666: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:13:25.695217372 (248h56m36.289328601s) [upload=239326]: Next returned nil segment Apr 18 01:13:25.695235968 (248h56m36.289347198s) [upload=239326]: Waiting for error group managing segments... Apr 18 01:13:25.857363625 (248h56m36.451474865s) [noise=true , address=145.79.227.13:4816 , piece_id=FC5OY5KCJ52DAHA7BWUHIFVQFARRN4W3ZQ7MQVF46EQIW7P2BWAA, seg_pos={0 0}, upload=239326]: Done uploading piece. err: <nil> Apr 18 01:13:25.921487662 (248h56m36.515598902s) [noise=true , address=145.79.227.32:4856 , piece_id=WS5BAD3VQMLAIK5ZDJPY6A5JIZ6OZMJELUAT22XYTQCVAKOH7MSA, seg_pos={0 0}, upload=239326]: Done uploading piece. err: <nil> Apr 18 01:13:26.389993243 (248h56m36.984104483s) [noise=true , address=193.149.164.71:4287 , piece_id=URCNFH2WDPJFP5JUFEQPFTUGCJIKJW7YMAZBEQG646J2QU5GVQLA, seg_pos={0 0}, upload=239326]: Done uploading piece. err: <nil> Apr 18 01:13:26.395049836 (248h56m36.989161076s) [noise=true , address=193.149.164.66:4986 , piece_id=OQ364K5WVAFPAZALKF2NPXZZRCJZDUX7NAADQXIFRSJXHCVSQD6Q, seg_pos={0 0}, upload=239326]: Done uploading piece. err: <nil> Apr 18 01:13:26.546936273 (248h56m37.141047517s) [noise=true , address=64.49.13.138:4483 , piece_id=CP24H3R353N7UHPBDJFBV7SHZW5ZLNHUXA727YSTWLFUUL2YH65Q, seg_pos={0 0}, upload=239326]: Done uploading piece. err: <nil> Apr 18 01:13:27.218602650 (248h56m37.812713893s) [noise=true , address=45.91.116.11:4140 , piece_id=AXB5HLQGR7PXL7YZK25NFRCC6D3FGQQPLS5BFX6ALFYKED5XFOOA, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:14DNa2ysTGtfy3Zz72Dq7S3NSGXqnTHZRkuXorDvuuBw1abwMT, address:45.91.116.11:4140): protocol: storage node overloaded, request limit: 2000 Apr 18 01:13:27.381132607 (248h56m37.975243847s) [noise=true , address=77.90.31.21:4905 , piece_id=4XJIVKGBZLMIESCCXODGBWN7VZQCXXRPAYF65NYLZNWEALFZEJMQ, seg_pos={0 0}, upload=239326]: Done uploading piece. err: <nil> Apr 18 01:13:29.496036837 (248h56m40.090148077s) [noise=true , address=145.79.227.21:4508 , piece_id=QSTJMOSCU5BODMDXRCU4MQXY2BJCEQGH6MMRGVAVKHSYA7432E2Q, seg_pos={0 0}, upload=239326]: Done uploading piece. err: <nil> Apr 18 01:14:24.718550736 (248h57m35.312661981s) [noise=true , address=145.79.227.33:4885 , piece_id=377XYUXSXT6TSSRG442UQRAULJTG7QKHJEJR5VJ6MDLVATU6NI5Q, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:14:24.719121868 (248h57m35.313233117s) [noise=true , address=145.79.227.43:4167 , piece_id=QDE3WXXUVBI6FHYI26BNUTJF5ZUXKUBWGF3KL6I5GCONWLEXJXKA, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:145.79.227.43:4167): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:14:24.719328361 (248h57m35.313439613s) [noise=true , address=193.149.164.143:4695 , piece_id=I6WAE2PQF75NNSZPPE7GMAS4OKEZNV2GT5YH5Z5GKRTZX7YMEP4A, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:12TZfgz24gnZr4UryzEKw6gu94njnmCvafX5X64MKFSJqaUWSVh, address:193.149.164.143:4695): protocol: write tcp 139.59.144.22:38692->193.149.164.143:4695: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:14:24.719414203 (248h57m35.313525453s) [noise=true , address=145.79.227.11:4761 , piece_id=WXKL24TUKQRGO6IBAEDH7VVZLZKCHYVNA2BEBSBKRDGH5B3A5HLQ, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:12CXHkaBwi15pBUoRiZHi6JP5rBDu1iVpWT1WFL3dCmk8LRRxqw, address:145.79.227.11:4761): protocol: rpc: dial tcp 145.79.227.11:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:14:24.719782828 (248h57m35.313894079s) [noise=true , address=167.17.72.70:4749 , piece_id=NVS3ARCT35NTLUOFK6BBWYWZ6KXP42TP4DAR64TN4IAGVD7FYFKA, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:1iawq6ZaX4YFCeunscwbuV4MNXfCPjPfDA4SVyUmtWHPVB6BdZ, address:167.17.72.70:4749): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:14:24.720042548 (248h57m35.314153805s) [noise=true , address=167.17.72.71:4789 , piece_id=AOQVFHXSNVWDDF5DSH56R2SFSXW7OOTP3NETVDNECQIFRNHM6ANQ, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:12AcmR5QqJktD68Ae2jycDEqL13se2yhT42JoVgDWMQCPCPbf9e, address:167.17.72.71:4789): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:14:24.720300459 (248h57m35.314411706s) [noise=true , address=145.79.227.40:4070 , piece_id=C47ABPSAFVDZ2G6PROWB7SAKBKIBIZLWWYLCLK5IJS3TWALDUS4Q, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me, address:145.79.227.40:4070): protocol: rpc: dial tcp 145.79.227.40:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:14:24.720409632 (248h57m35.314520884s) [noise=true , address=145.79.227.38:4993 , piece_id=X5AWTFDB3JG2WGGKSMWZLXUYHB2I6LKH6GMR262D7NCGPHZQOH7Q, seg_pos={0 0}, upload=239326]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: read tcp 139.59.144.22:54530->145.79.227.38:4993: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:14:24.720443489 (248h57m35.314554729s) [seg_pos={0 0}, upload=239326]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Apr 18 01:14:24.822937780 (248h57m35.417049009s) [upload=239326]: Done uploading segments... Apr 18 01:14:24.822976045 (248h57m35.417087287s) [upload=239326]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.106708799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXXXXXXX] duration: 5.001478931s trace: https://jaeger.dev.storj.io/trace/58ac5fca5168134a FAIL (2026-04-18 01:13:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001505617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..X..X..X..X....X....X.....X....XX..X..........XX...XX.XXX.X] duration: 1m0.001257292s trace: https://jaeger.dev.storj.io/trace/44e3972bbc202550 FAIL (2026-04-18 01:13:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001299077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 46 out of 60 is failed (threshold=3) [.............XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.109269143s trace: https://jaeger.dev.storj.io/trace/592ef4268b2cddea FAIL (2026-04-18 01:13:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== _pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.858796863 (248h55m35.452908114s) [noise=true , address=167.17.72.69:4734 , piece_id=TMYNLIS27MUOKVQW6IFDBOXHK33LTETHPEGX36MKUDNSY4KB3LSQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.859142666 (248h55m35.453253914s) [noise=true , address=145.79.227.2:4120 , piece_id=BFK27NMH75B3FQZEP3KRAPDLY6SOOHIXIJHF5UCETCEFEQIDWPNQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.859798668 (248h55m35.453909919s) [noise=true , address=45.91.116.14:4201 , piece_id=FLZSIFPLHOESWPMJ2HKCJEFWQUEVGPPCBFRQDXZZHP5J5HDT47MA, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.860422683 (248h55m35.454533937s) [noise=true , address=145.79.227.29:4958 , piece_id=KUDMEYPBALCHYR4Q2ISYTPBUFE7TQ2HYNRX4F6NBFZOX24BLEPLQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.860939198 (248h55m35.455050446s) [noise=true , address=145.79.227.34:4933 , piece_id=BVRZ7CW6EJATPAT4PZVPJ4YUJMO3IAF2CBCYSQIOBCXLS23HQTFQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.861362289 (248h55m35.455473536s) [noise=true , address=193.149.164.66:4973 , piece_id=2AVFL3CMLKQ4CEFR234TMNWL2ESSRQVGP3POXOAVGXSPUPSSIXOQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.861807367 (248h55m35.455918615s) [noise=true , address=77.90.31.12:4617 , piece_id=OFBLNLPDAP67B2QD4XT7BSWRQPCTMGQVIIDRF67GUOAU7WFXT7MA, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.862281727 (248h55m35.456392971s) [noise=true , address=145.79.227.28:4759 , piece_id=XD4HZI2A7LYC3IZMJ6QCDDCBKSXX36CNVKEK6MWNBQUKZBGVNKPQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.862728986 (248h55m35.456840228s) [noise=true , address=193.149.164.71:4295 , piece_id=OQ2JXUCEUYGOR2JYNA5C2LMKRGE4HRSJ636XV6PP5FUQGKKD6QEQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.863541730 (248h55m35.457652981s) [noise=true , address=145.79.227.7:4326 , piece_id=V2SWF3OKGOELKK2B4HPEPPWWE6NAST65SMWWPJYYGPZX4D2H5AKQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.864277267 (248h55m35.458388522s) [noise=true , address=167.17.72.71:4791 , piece_id=BS27EPH3AVW7OATWXLOVG77P3GKIMJYVC3RAWKOMEEG6IWEWAHEA, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.864959913 (248h55m35.459071166s) [noise=true , address=145.79.227.11:4761 , piece_id=4PWUCMY7LWSXITC5H6ZOGGQYQP2VANX2NYRZCLCDLQZ647FX5OAQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.865562941 (248h55m35.459674194s) [noise=true , address=193.149.164.143:4701 , piece_id=RKBKWFU526QO44KVFTO5GQ6IAESMG6HZI7AZESIV7B5OQB3ZXJEQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.866227368 (248h55m35.460338621s) [noise=true , address=45.91.116.11:4140 , piece_id=RCDBPONN7A3N2HNY5MK2ZPHZOK3JDTCX7L36PFCCIF5DEL4HIBCQ, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.866874685 (248h55m35.460985913s) [noise=true , address=145.79.227.38:4987 , piece_id=3DMSR4U7U6R36OYSPDWHTFCGZLBEDQYWXPWJYZMZHQ7U6RWY7EQA, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:24.870937472 (248h55m35.465048702s) [noise=true , address=167.17.72.68:4650 , piece_id=2KC3EXOUBAWEEGTIUDI7Y5LKN2KR2L2SFHQNX6O77TE47DT5TINA, seg_pos={0 0}, upload=239311]: Uploading piece... Apr 18 01:12:25.013756191 (248h55m35.607867439s) [noise=true , address=167.17.72.68:4650 , piece_id=2KC3EXOUBAWEEGTIUDI7Y5LKN2KR2L2SFHQNX6O77TE47DT5TINA, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:12VnVRKqd9wzV3r4DiR9KvbFPPxssAE2VudYYDFvgWtmAuVdGN6, address:167.17.72.68:4650): protocol: write tcp 139.59.144.22:52620->167.17.72.68:4650: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:12:25.322837712 (248h55m35.916948974s) [noise=true , address=77.90.31.12:4617 , piece_id=OFBLNLPDAP67B2QD4XT7BSWRQPCTMGQVIIDRF67GUOAU7WFXT7MA, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:1PpGinvTbgS3cMHoLaMhhyab7L3urApr3DJRJG4SMzWqw8UAPg, address:77.90.31.12:4617): protocol: storage node overloaded, request limit: 2000 Apr 18 01:12:25.452682899 (248h55m36.046794156s) [noise=true , address=167.17.72.71:4791 , piece_id=BS27EPH3AVW7OATWXLOVG77P3GKIMJYVC3RAWKOMEEG6IWEWAHEA, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:1nJkNrQsUMSuCHPpZkW5ztce4Y6hTe2CbtAhBDqGtHn6mQrc1f, address:167.17.72.71:4791): protocol: storage node overloaded, request limit: 2000 Apr 18 01:12:25.453776300 (248h55m36.047887543s) [upload=239311]: Next returned nil segment Apr 18 01:12:25.453794154 (248h55m36.047905403s) [upload=239311]: Waiting for error group managing segments... Apr 18 01:12:25.604657396 (248h55m36.198768696s) [noise=true , address=145.79.227.28:4759 , piece_id=XD4HZI2A7LYC3IZMJ6QCDDCBKSXX36CNVKEK6MWNBQUKZBGVNKPQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: <nil> Apr 18 01:12:25.728118042 (248h55m36.322229286s) [noise=true , address=45.91.116.14:4201 , piece_id=FLZSIFPLHOESWPMJ2HKCJEFWQUEVGPPCBFRQDXZZHP5J5HDT47MA, seg_pos={0 0}, upload=239311]: Done uploading piece. err: <nil> Apr 18 01:12:26.104865431 (248h55m36.698976674s) [noise=true , address=193.149.164.71:4295 , piece_id=OQ2JXUCEUYGOR2JYNA5C2LMKRGE4HRSJ636XV6PP5FUQGKKD6QEQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: <nil> Apr 18 01:12:26.122989568 (248h55m36.717100818s) [noise=true , address=193.149.164.66:4973 , piece_id=2AVFL3CMLKQ4CEFR234TMNWL2ESSRQVGP3POXOAVGXSPUPSSIXOQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: <nil> Apr 18 01:12:27.105143417 (248h55m37.699254674s) [noise=true , address=45.91.116.11:4140 , piece_id=RCDBPONN7A3N2HNY5MK2ZPHZOK3JDTCX7L36PFCCIF5DEL4HIBCQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: <nil> Apr 18 01:12:30.950689966 (248h55m41.544801212s) [noise=true , address=145.79.227.7:4326 , piece_id=V2SWF3OKGOELKK2B4HPEPPWWE6NAST65SMWWPJYYGPZX4D2H5AKQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: <nil> Apr 18 01:12:42.406192424 (248h55m53.000303656s) [noise=true , address=145.79.227.2:4120 , piece_id=BFK27NMH75B3FQZEP3KRAPDLY6SOOHIXIJHF5UCETCEFEQIDWPNQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: <nil> Apr 18 01:13:24.452958227 (248h56m35.047069478s) [noise=true , address=193.149.164.143:4701 , piece_id=RKBKWFU526QO44KVFTO5GQ6IAESMG6HZI7AZESIV7B5OQB3ZXJEQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz, address:193.149.164.143:4701): protocol: write tcp 139.59.144.22:36236->193.149.164.143:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:13:24.452986988 (248h56m35.047098237s) [noise=true , address=167.17.72.69:4734 , piece_id=TMYNLIS27MUOKVQW6IFDBOXHK33LTETHPEGX36MKUDNSY4KB3LSQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:12bGaxiZQ1uHmbYMfJ273PHh1jni4AVtTjcBEM38ZFYm2j1rV5g, address:167.17.72.69:4734): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:13:24.453392280 (248h56m35.04750353s) [noise=true , address=145.79.227.34:4933 , piece_id=BVRZ7CW6EJATPAT4PZVPJ4YUJMO3IAF2CBCYSQIOBCXLS23HQTFQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou, address:145.79.227.34:4933): protocol: rpc: dial tcp 145.79.227.34:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:13:24.453521304 (248h56m35.047632553s) [noise=true , address=145.79.227.29:4958 , piece_id=KUDMEYPBALCHYR4Q2ISYTPBUFE7TQ2HYNRX4F6NBFZOX24BLEPLQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:1ySYdoPKsX8RRcnfGuneBqVjXLbuDQJ7ofRiv6dEyYxR32hoGR, address:145.79.227.29:4958): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:13:24.453709091 (248h56m35.04782034s) [noise=true , address=145.79.227.38:4987 , piece_id=3DMSR4U7U6R36OYSPDWHTFCGZLBEDQYWXPWJYZMZHQ7U6RWY7EQA, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:1oZxiaBQD1grLu8d1HyERweeuXNXc1aBgyAoe1B86gqm6ypvBC, address:145.79.227.38:4987): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:13:24.453892537 (248h56m35.048003785s) [noise=true , address=145.79.227.11:4761 , piece_id=4PWUCMY7LWSXITC5H6ZOGGQYQP2VANX2NYRZCLCDLQZ647FX5OAQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:12CXHkaBwi15pBUoRiZHi6JP5rBDu1iVpWT1WFL3dCmk8LRRxqw, address:145.79.227.11:4761): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:13:24.454207286 (248h56m35.048318537s) [noise=true , address=145.79.227.33:4875 , piece_id=CULEFWJHWCNEETI7WTOS5IK4F5R3VWM2KVGDJLSEXY2IW6GHX2PQ, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:13:24.454296792 (248h56m35.04840804s) [noise=true , address=145.79.227.40:4057 , piece_id=Y35HBFQI27UGLGJLVUQACIE3ZO2NF547SLR4WUMX4BKRFSFZMIJA, seg_pos={0 0}, upload=239311]: Done uploading piece. err: ecclient: upload failed (node:12Ja2NPTrk5uh5ofH3TMvPpwk1S789UX3L35K8TMwj2uNn2223P, address:145.79.227.40:4057): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:13:24.454327533 (248h56m35.048438762s) [seg_pos={0 0}, upload=239311]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Apr 18 01:13:24.558236283 (248h56m35.152347532s) [upload=239311]: Done uploading segments... Apr 18 01:13:24.558282276 (248h56m35.152393527s) [upload=239311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.109286539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXXXXXX] duration: 5.001292718s trace: https://jaeger.dev.storj.io/trace/4694ea675bdd1c06 FAIL (2026-04-18 01:12:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001317945s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 45 out of 60 is failed (threshold=3) [..............XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.108923234s trace: https://jaeger.dev.storj.io/trace/9f54bf6446f9d79 FAIL (2026-04-18 01:12:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== e , address=167.17.72.71:4799 , piece_id=4DURHZ37N7UZU3FUTELRYP6TXME277VOOMLFQKXM6ED75NCQ6A5Q, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.566141930 (248h54m35.16025318s) [noise=true , address=45.91.116.10:4105 , piece_id=QUWO2NCOSQ22GFDUNNGQSDHCPHPDXGEQBO33JKVGL3H7KHC4LJIQ, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.566636998 (248h54m35.160748243s) [noise=true , address=145.79.227.2:4120 , piece_id=LYDKEAD5IASPVJYMT2RJ6MPF7L753JCYZYHTYLSKZCZBNY4YBDHA, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.566781575 (248h54m35.160892814s) [noise=true , address=167.17.72.70:4764 , piece_id=QQ6CJSBPOCDH272ROG374TJEJVSNODGBUWKRZOTVEBQUEAL2WADQ, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.566940250 (248h54m35.161051495s) [noise=true , address=167.17.72.68:4643 , piece_id=F4YAZILCNNAUMJAOZIBZR75IOQMGMRPOJSJR5TPHDXQFLKW36GYA, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.567225067 (248h54m35.16133631s) [noise=true , address=77.90.31.16:4764 , piece_id=TXFMA7YV3YRFBH5XPPIGO65EQ5OSLSDTHXGJJQ2ISE7KB5FFRUIA, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.567276818 (248h54m35.16138806s) [noise=true , address=77.90.31.19:4863 , piece_id=5DKABI3FJU7QBLQAJQNOQOCEBLOAJAAMW6UHUORYIBMRNE4GVN4A, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.567528961 (248h54m35.161640199s) [noise=true , address=145.79.227.41:4099 , piece_id=IB4TSK774REBGUEREM5ANWITTMXCRGVQG3SWLEK3J5MFKRZBLB5A, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.567751459 (248h54m35.161862702s) [noise=true , address=77.90.31.18:4814 , piece_id=55KCJNDW2ZQO3ZOURTFBQFLACZZSFPPFDAARZV54IOLNY3AKSHHQ, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.567803740 (248h54m35.161914982s) [noise=true , address=193.149.164.66:4962 , piece_id=2J3HXGIUAL4CHQQJRH63BPMJUUPEVXCFOHV5VI3ALCDJCY7TDHBA, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.568083268 (248h54m35.16219451s) [noise=true , address=167.17.72.69:4732 , piece_id=EZMZH53NBGCC275NZ6U3J4TWNVNMBTVVP255H2DPXYD3C23YFM2Q, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.568207193 (248h54m35.162318443s) [noise=true , address=145.79.227.31:4809 , piece_id=MCJWCYNWSLPPCT2SJ7N7HOLEBNLFGOOYMH7E44VG2ZEAPJDUSIFA, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.568718109 (248h54m35.162829354s) [noise=true , address=145.79.227.11:4758 , piece_id=KNFCT4VVLBJSUDPKXKZ5M5QCVLS4VWMDOLBGO4XSQCEWI7SWOV7Q, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.569228512 (248h54m35.163339752s) [noise=true , address=77.90.31.17:4794 , piece_id=UMWQVRKHWRZDO5VJ3ORVAZGXPQZVFHWFVGCBXPVKSAH2AADX76GA, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.569767635 (248h54m35.163878883s) [noise=true , address=145.79.227.21:4508 , piece_id=L24NCL5AQLMNXCFYPNQAQNV2AJQHDCLT6MQQ2Q6WTJZDAPG7QI5Q, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.570307884 (248h54m35.16441913s) [noise=true , address=145.79.227.34:4927 , piece_id=YBU7K3OE5VBSTA42M23EEHQ3DQCVYYORFSALPOGCSRYRCREC74NA, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:24.570774429 (248h54m35.164885677s) [noise=true , address=77.90.31.10:4536 , piece_id=5X4KAJASXNKHTHDQIAFOO5YX7RCD6ZYG25HKANDJZMNLINMYGKEA, seg_pos={0 0}, upload=239297]: Uploading piece... Apr 18 01:11:25.135881388 (248h54m35.729992646s) [upload=239297]: Next returned nil segment Apr 18 01:11:25.135900560 (248h54m35.730011818s) [upload=239297]: Waiting for error group managing segments... Apr 18 01:11:25.358119100 (248h54m35.952230354s) [noise=true , address=145.79.227.41:4099 , piece_id=IB4TSK774REBGUEREM5ANWITTMXCRGVQG3SWLEK3J5MFKRZBLB5A, seg_pos={0 0}, upload=239297]: Done uploading piece. err: <nil> Apr 18 01:11:25.778830659 (248h54m36.372941903s) [noise=true , address=77.90.31.10:4536 , piece_id=5X4KAJASXNKHTHDQIAFOO5YX7RCD6ZYG25HKANDJZMNLINMYGKEA, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:1zwuzCLWy2q166Ld52WUG9fvXNdJWsQrg2NearHYg7pyxpXFen, address:77.90.31.10:4536): protocol: storage node overloaded, request limit: 2000 Apr 18 01:11:26.222158572 (248h54m36.816269831s) [noise=true , address=193.149.164.66:4962 , piece_id=2J3HXGIUAL4CHQQJRH63BPMJUUPEVXCFOHV5VI3ALCDJCY7TDHBA, seg_pos={0 0}, upload=239297]: Done uploading piece. err: <nil> Apr 18 01:11:26.683083337 (248h54m37.277194592s) [noise=true , address=77.90.31.16:4764 , piece_id=TXFMA7YV3YRFBH5XPPIGO65EQ5OSLSDTHXGJJQ2ISE7KB5FFRUIA, seg_pos={0 0}, upload=239297]: Done uploading piece. err: <nil> Apr 18 01:11:26.878439929 (248h54m37.472551177s) [noise=true , address=145.79.227.21:4508 , piece_id=L24NCL5AQLMNXCFYPNQAQNV2AJQHDCLT6MQQ2Q6WTJZDAPG7QI5Q, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:145.79.227.21:4508): protocol: storage node overloaded, request limit: 2000 Apr 18 01:11:27.209855919 (248h54m37.803967168s) [noise=true , address=77.90.31.19:4863 , piece_id=5DKABI3FJU7QBLQAJQNOQOCEBLOAJAAMW6UHUORYIBMRNE4GVN4A, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:124x1cbCPJRK8YkgSP6UzHymoLu1E1znnfg5WBiD89kSDLi8JeL, address:77.90.31.19:4863): protocol: storage node overloaded, request limit: 2000 Apr 18 01:11:28.935239069 (248h54m39.529350309s) [noise=true , address=45.91.116.10:4105 , piece_id=QUWO2NCOSQ22GFDUNNGQSDHCPHPDXGEQBO33JKVGL3H7KHC4LJIQ, seg_pos={0 0}, upload=239297]: Done uploading piece. err: <nil> Apr 18 01:11:30.321505587 (248h54m40.915616828s) [noise=true , address=145.79.227.31:4809 , piece_id=MCJWCYNWSLPPCT2SJ7N7HOLEBNLFGOOYMH7E44VG2ZEAPJDUSIFA, seg_pos={0 0}, upload=239297]: Done uploading piece. err: <nil> Apr 18 01:11:33.720238563 (248h54m44.31434982s) [noise=true , address=145.79.227.2:4120 , piece_id=LYDKEAD5IASPVJYMT2RJ6MPF7L753JCYZYHTYLSKZCZBNY4YBDHA, seg_pos={0 0}, upload=239297]: Done uploading piece. err: <nil> Apr 18 01:11:34.950445417 (248h54m45.544556663s) [noise=true , address=77.90.31.17:4794 , piece_id=UMWQVRKHWRZDO5VJ3ORVAZGXPQZVFHWFVGCBXPVKSAH2AADX76GA, seg_pos={0 0}, upload=239297]: Done uploading piece. err: <nil> Apr 18 01:11:46.039163686 (248h54m56.633274942s) [noise=true , address=77.90.31.18:4814 , piece_id=55KCJNDW2ZQO3ZOURTFBQFLACZZSFPPFDAARZV54IOLNY3AKSHHQ, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: storage node overloaded, request limit: 2000 Apr 18 01:12:24.155404737 (248h55m34.749515984s) [noise=true , address=167.17.72.68:4643 , piece_id=F4YAZILCNNAUMJAOZIBZR75IOQMGMRPOJSJR5TPHDXQFLKW36GYA, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:12bko3GBMbLQw4f4LgvJnLnSngLpGLAqMjGBiN9HRfFjsixjTE7, address:167.17.72.68:4643): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:12:24.155492940 (248h55m34.749604189s) [noise=true , address=167.17.72.70:4764 , piece_id=QQ6CJSBPOCDH272ROG374TJEJVSNODGBUWKRZOTVEBQUEAL2WADQ, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:1RV9Ee1uyE4ENr7frhwbKi2Ai2PnK54QrDWebuhyaRnWwdVk59, address:167.17.72.70:4764): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:12:24.155706422 (248h55m34.749817669s) [noise=true , address=167.17.72.71:4799 , piece_id=4DURHZ37N7UZU3FUTELRYP6TXME277VOOMLFQKXM6ED75NCQ6A5Q, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:1DRqc96ZCXVGKcihB41A6cYaN686EoS5QqVJWdqTddy7FroHy4, address:167.17.72.71:4799): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:12:24.155880945 (248h55m34.749992193s) [noise=true , address=145.79.227.11:4758 , piece_id=KNFCT4VVLBJSUDPKXKZ5M5QCVLS4VWMDOLBGO4XSQCEWI7SWOV7Q, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:1joBbXdmusUXDfe1ofLp5oXy1nVkkRAng8QGL2hAtKjDz71u5g, address:145.79.227.11:4758): protocol: rpc: dial tcp 145.79.227.11:4758: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:12:24.155955006 (248h55m34.750066251s) [noise=true , address=167.17.72.69:4732 , piece_id=EZMZH53NBGCC275NZ6U3J4TWNVNMBTVVP255H2DPXYD3C23YFM2Q, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:1DHibn2sd62kA6nJd9mHV6LgoKD43kQsKbbTwRSmiaQrvSHHB6, address:167.17.72.69:4732): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:12:24.156280984 (248h55m34.750392235s) [noise=true , address=145.79.227.33:4885 , piece_id=6OMWUNTD5RWTFRTNOO3YVGRZQMFQ5OIO5YWFHTN6VSLQQ772MZKQ, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG, address:145.79.227.33:4885): protocol: rpc: dial tcp 145.79.227.33:4885: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:12:24.156423880 (248h55m34.750535121s) [noise=true , address=145.79.227.34:4927 , piece_id=YBU7K3OE5VBSTA42M23EEHQ3DQCVYYORFSALPOGCSRYRCREC74NA, seg_pos={0 0}, upload=239297]: Done uploading piece. err: ecclient: upload failed (node:121bjMnf34sycS3W4KZpjVXCxwdmktaYj3rkKT3XqntTa18NKsm, address:145.79.227.34:4927): protocol: rpc: dial tcp 145.79.227.34:4927: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:12:24.156578127 (248h55m34.750689356s) [seg_pos={0 0}, upload=239297]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Apr 18 01:12:24.260841339 (248h55m34.854952586s) [upload=239297]: Done uploading segments... Apr 18 01:12:24.260903902 (248h55m34.855015152s) [upload=239297]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.108945124s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [...X..X..X..X....X....X.....X....XX..X..........XX...XX.XXX.] duration: 7.187522342s trace: https://jaeger.dev.storj.io/trace/5ed15fd658562d2e FAIL (2026-04-18 01:12:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.187551126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [....X..X..X..X....X....X.....X....XX..X..........XX...XX.XXX] duration: 1m0.001139637s trace: https://jaeger.dev.storj.io/trace/5ed15fd658562d2e FAIL (2026-04-18 01:11:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001154712s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXXXXX] duration: 5.001256441s trace: https://jaeger.dev.storj.io/trace/324c27f7afd61434 FAIL (2026-04-18 01:11:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001343052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 44 out of 60 is failed (threshold=3) [...............XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.114355853s trace: https://jaeger.dev.storj.io/trace/1085ded730d2afab FAIL (2026-04-18 01:11:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== 8h53m34.882426465s) [noise=true , address=167.17.72.69:4733 , piece_id=OZ4PL2RBXUDQVVLU4PAGCKBI6JNZJEZCYCHMK5KDDSBFR4VRZNDA, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.288666684 (248h53m34.882777938s) [noise=true , address=145.79.227.31:4810 , piece_id=XVLUR2E4URQDD7VOAWCQXNPEBJ3RGRZ2QSCGTTGE5L6BS5FX6INQ, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.288740445 (248h53m34.882851692s) [noise=true , address=167.17.72.68:4645 , piece_id=AAELNYCBCDV5YTQCPDUBW2O2WZU7SF7EFJRRIVSKZ235AOSD4S4A, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.289102582 (248h53m34.883213831s) [noise=true , address=77.90.31.19:4865 , piece_id=NANJ3B3EOXZTQ3ISTHQEN7JOBBIFFANSNOO7WRBC2XVIU3MRTFFQ, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.289129963 (248h53m34.883241213s) [noise=true , address=45.91.116.11:4130 , piece_id=MMYSTHJ7ONE5DGIRM3DSPBPZZUF54P7OBRFXV4KWEDJQL6ESDMBQ, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.289549814 (248h53m34.883661064s) [noise=true , address=145.79.227.49:4360 , piece_id=PKYHXFR6R4RETY2GZIXS5QH3HCICTLK5DKQHL6QLNOMXEDYYIN5A, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.289578100 (248h53m34.883689342s) [noise=true , address=167.17.72.71:4782 , piece_id=62E6JWWJKZRTBZPG37HVDUAX35WXAZTFM3TSYC762MJAZV4WAV3Q, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.289999012 (248h53m34.884110254s) [noise=true , address=167.17.72.70:4773 , piece_id=B34KUCZBDFO2HHSIXP5K5RC3ZPP62YIKLNMR5VY42TFHRRYA7RUQ, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.290122387 (248h53m34.884233636s) [noise=true , address=193.149.164.66:4990 , piece_id=6CGXVGXWCEFYXPLK7BTS5IYO7BL3S42MEV4CUTYPDSEALCQO3TTA, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.290517988 (248h53m34.88462962s) [noise=true , address=193.149.164.143:4694 , piece_id=5RAP5X63AVEBTBGD5XDB2SZ5SVCNSF6HABS6LEAYYAT2257CKS4A, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.290908167 (248h53m34.885019417s) [noise=true , address=145.79.227.29:4971 , piece_id=N7S5ZTJFAJE24AZLJNG5P5JJWK7XYDK3TJWZPYGXC2SJZJPHNRVA, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.291336400 (248h53m34.88544765s) [noise=true , address=193.149.164.71:4296 , piece_id=5N6RFI3OZOFHRVORITCF75TXIJUJ3DMB65ZJ5AZJOAJX7MKXC4KA, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.291739119 (248h53m34.885850369s) [noise=true , address=145.79.227.13:4828 , piece_id=5H7LA2L7DBG5KC35J4UBHFU4SNMXAD465K5W2OFXG6LNGBFREMXA, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.292150497 (248h53m34.886261745s) [noise=true , address=145.79.227.7:4326 , piece_id=LSYWRUI6FOZM36C7NA5R27NVNGCWDPNYRXHTFULUZXQ3QMI72OQQ, seg_pos={0 0}, upload=239279]: Uploading piece... Apr 18 01:10:24.378492629 (248h53m34.97260388s) [noise=true , address=145.79.227.49:4360 , piece_id=PKYHXFR6R4RETY2GZIXS5QH3HCICTLK5DKQHL6QLNOMXEDYYIN5A, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: rpc: dial tcp 145.79.227.49:4360: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:10:24.436880113 (248h53m35.030991363s) [noise=true , address=167.17.72.71:4782 , piece_id=62E6JWWJKZRTBZPG37HVDUAX35WXAZTFM3TSYC762MJAZV4WAV3Q, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:12dMYQiMeF5xy8HRmC7pCRJqkRDDHX3LKHKdVrMKYTFEVYPPWCC, address:167.17.72.71:4782): protocol: write tcp 139.59.144.22:55826->167.17.72.71:4782: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:10:24.678696104 (248h53m35.272807356s) [noise=true , address=145.79.227.12:4780 , piece_id=CCDVWXHV6OKFZMPO7CSWJV4DZSDDSXTRRFRUZPA3M2GCKMYNJ4AQ, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:1i8cJRxzLNvUYqELrufixk24oDZwDHroyG5Cfy13csTPgeajRM, address:145.79.227.12:4780): protocol: storage node overloaded, request limit: 2000 Apr 18 01:10:24.875901988 (248h53m35.470013219s) [upload=239279]: Next returned nil segment Apr 18 01:10:24.875919174 (248h53m35.470030401s) [upload=239279]: Waiting for error group managing segments... Apr 18 01:10:25.185512570 (248h53m35.779623816s) [noise=true , address=77.90.31.19:4865 , piece_id=NANJ3B3EOXZTQ3ISTHQEN7JOBBIFFANSNOO7WRBC2XVIU3MRTFFQ, seg_pos={0 0}, upload=239279]: Done uploading piece. err: <nil> Apr 18 01:10:25.314249649 (248h53m35.908360891s) [noise=true , address=145.79.227.13:4828 , piece_id=5H7LA2L7DBG5KC35J4UBHFU4SNMXAD465K5W2OFXG6LNGBFREMXA, seg_pos={0 0}, upload=239279]: Done uploading piece. err: <nil> Apr 18 01:10:25.319122969 (248h53m35.91323422s) [noise=true , address=145.79.227.7:4326 , piece_id=LSYWRUI6FOZM36C7NA5R27NVNGCWDPNYRXHTFULUZXQ3QMI72OQQ, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:12pU8MZZKS88CPFdZoaQ4msNfkCbCewTmS2wKEdN7pG5aCWoAr6, address:145.79.227.7:4326): protocol: storage node overloaded, request limit: 2000 Apr 18 01:10:25.590856124 (248h53m36.184967363s) [noise=true , address=193.149.164.71:4296 , piece_id=5N6RFI3OZOFHRVORITCF75TXIJUJ3DMB65ZJ5AZJOAJX7MKXC4KA, seg_pos={0 0}, upload=239279]: Done uploading piece. err: <nil> Apr 18 01:10:28.511751944 (248h53m39.105863183s) [noise=true , address=45.91.116.11:4130 , piece_id=MMYSTHJ7ONE5DGIRM3DSPBPZZUF54P7OBRFXV4KWEDJQL6ESDMBQ, seg_pos={0 0}, upload=239279]: Done uploading piece. err: <nil> Apr 18 01:10:32.095333326 (248h53m42.689444576s) [noise=true , address=145.79.227.31:4810 , piece_id=XVLUR2E4URQDD7VOAWCQXNPEBJ3RGRZ2QSCGTTGE5L6BS5FX6INQ, seg_pos={0 0}, upload=239279]: Done uploading piece. err: <nil> Apr 18 01:10:32.345565208 (248h53m42.939676446s) [noise=true , address=145.79.227.37:4573 , piece_id=RPAM2PZC3EUELJDCYJC5OAM2LIR7LWNIJKSTIAU4PJ5F2JJZ4A6A, seg_pos={0 0}, upload=239279]: Done uploading piece. err: <nil> Apr 18 01:11:23.882675949 (248h54m34.476787206s) [noise=true , address=145.79.227.11:4750 , piece_id=IBYVPN6Z6ECM7YL46CJ2FLHAMLKKG4ZCIYVDCF5CLF5EIBI3WB5A, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:12XoDDL5HwunayYqgQDzvxMsgFnTx1kUU6aiyvhsRLucnR9eTUz, address:145.79.227.11:4750): protocol: rpc: dial tcp 145.79.227.11:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:11:23.882780202 (248h54m34.476891451s) [noise=true , address=145.79.227.34:4912 , piece_id=PISP3FT7SI77VMCETFASK2RULXWOWLNYFSMF2IRGKX4DIZLNJWXQ, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: rpc: dial tcp 145.79.227.34:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:11:23.883075308 (248h54m34.477186556s) [noise=true , address=167.17.72.69:4733 , piece_id=OZ4PL2RBXUDQVVLU4PAGCKBI6JNZJEZCYCHMK5KDDSBFR4VRZNDA, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:12RqZJBPPTLz3JQAniTZm5HksoCLmci9XStzU7qbUmLfm2M7Jy3, address:167.17.72.69:4733): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:11:23.883274004 (248h54m34.477385255s) [noise=true , address=167.17.72.68:4645 , piece_id=AAELNYCBCDV5YTQCPDUBW2O2WZU7SF7EFJRRIVSKZ235AOSD4S4A, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:12LkJ3iu6FoMMpqkwtLrD9Q5zTrwk7s57bPtJzRw1kgzXTih6kE, address:167.17.72.68:4645): protocol: write tcp 139.59.144.22:50158->167.17.72.68:4645: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:11:23.883353097 (248h54m34.477464339s) [noise=true , address=193.149.164.143:4694 , piece_id=5RAP5X63AVEBTBGD5XDB2SZ5SVCNSF6HABS6LEAYYAT2257CKS4A, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, address:193.149.164.143:4694): protocol: write tcp 139.59.144.22:50824->193.149.164.143:4694: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:11:23.883525140 (248h54m34.477636391s) [noise=true , address=145.79.227.29:4971 , piece_id=N7S5ZTJFAJE24AZLJNG5P5JJWK7XYDK3TJWZPYGXC2SJZJPHNRVA, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH, address:145.79.227.29:4971): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:11:23.883542396 (248h54m34.477653648s) [noise=true , address=167.17.72.70:4773 , piece_id=B34KUCZBDFO2HHSIXP5K5RC3ZPP62YIKLNMR5VY42TFHRRYA7RUQ, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:1yGCQR1uoPNNAvA5PhhhzmHbxxu1YdrUrSa6cTtPHHo8rywx7Z, address:167.17.72.70:4773): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:11:23.883788900 (248h54m34.477900141s) [noise=true , address=193.149.164.66:4990 , piece_id=6CGXVGXWCEFYXPLK7BTS5IYO7BL3S42MEV4CUTYPDSEALCQO3TTA, seg_pos={0 0}, upload=239279]: Done uploading piece. err: ecclient: upload failed (node:12rZoTqAdgDDLWrSLga3M8CKSCNayUX33YFx5HB6g1FDjrVHUcq, address:193.149.164.66:4990): protocol: write tcp 139.59.144.22:48670->193.149.164.66:4990: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:11:23.883825439 (248h54m34.477936669s) [seg_pos={0 0}, upload=239279]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Apr 18 01:11:23.992761716 (248h54m34.586872958s) [upload=239279]: Done uploading segments... Apr 18 01:11:23.992811546 (248h54m34.586922795s) [upload=239279]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.1143737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXXXX] duration: 5.001322364s trace: https://jaeger.dev.storj.io/trace/7760f31ebddd86f1 FAIL (2026-04-18 01:10:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00134671s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [X....X..X..X..X....X....X.....X....XX..X..........XX...XX.XX] duration: 1m0.000972698s trace: https://jaeger.dev.storj.io/trace/66719410c0287cd3 FAIL (2026-04-18 01:10:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000986946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 43 out of 60 is failed (threshold=3) [................XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.107477594s trace: https://jaeger.dev.storj.io/trace/487ef73b5d78847a FAIL (2026-04-18 01:10:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== d=239266]: Uploading piece... Apr 18 01:09:24.019927238 (248h52m34.614038491s) [noise=true , address=145.79.227.18:4019 , piece_id=PFMRRQGGTIEFMDPUOIC45NJ6CBIQW3O6TSCCCKRYDNXZGLBU5IBA, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.020256573 (248h52m34.614367824s) [noise=true , address=145.79.227.11:4744 , piece_id=ZIRG5LCWV3VJWB23IDNI6XZOPID7O7R75JNMJ3R2AGSBFTER5MXA, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.020466595 (248h52m34.614577842s) [noise=true , address=167.17.72.71:4794 , piece_id=4TB7C3COXSZFBLU5OTZYEGQ3JLMI3ZTIDFK67KVPRSXI4ZTH4AZA, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.020890949 (248h52m34.615002198s) [noise=true , address=145.79.227.21:4508 , piece_id=FRFW2WYAZY4XXYWUCYKVVXYE6C2P47PRNH2FNCTRO4VKF5FEHVAA, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.021087102 (248h52m34.615198356s) [noise=true , address=77.90.31.18:4814 , piece_id=TMW4O7IBV5F6MTVG7TLW57446PVU44ZJSCXP2GQGDQSIVGBGP6RA, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.021305079 (248h52m34.615416328s) [noise=true , address=145.79.227.29:4949 , piece_id=FS2FG4NBMFUUXGKJ5TPINN5PBEIGT4KBZLN7OSHQMLGEE23TNSHQ, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.021763607 (248h52m34.615874858s) [noise=true , address=64.49.13.140:4513 , piece_id=KPD55IU3GPNS7ASRYLLELY6TAPO5HMM22HA5XCADYFJQV6YPACIA, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.022821635 (248h52m34.616932891s) [noise=true , address=77.90.31.17:4792 , piece_id=73QIZUS4FT22OOQY2I5EAJZ5QGVM375M7K75AZYYTTXW6R2BQJHQ, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.023672151 (248h52m34.617783408s) [noise=true , address=193.149.164.143:4702 , piece_id=Z4DIMQQAAKIVYBU5A25UITI7KQO3XT6XNLUYV3RYMSBLGMVHUAEQ, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.025612298 (248h52m34.619723547s) [noise=true , address=145.79.227.34:4933 , piece_id=EJZJ7WUCVTNHB72JCSXDL7D3O7TFUYDJU5SBSZPME3KFRHTKCGCA, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.026884941 (248h52m34.620996221s) [noise=true , address=167.17.72.69:4737 , piece_id=Q6HNZEMZDYPXGF2HWFENCTF24E2N6L4BYXNNWPWKX7OCV7E3O5HQ, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.027704984 (248h52m34.621816238s) [noise=true , address=64.49.13.169:4706 , piece_id=MWUTUT6HQMBCKX7OCJMNADB3RSVOFR5AHTP6GQ4B4DDU5C7WQVAQ, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.028352282 (248h52m34.622463534s) [noise=true , address=45.91.116.13:4180 , piece_id=5R2ZNYAJTRY4KGYH2SW67C4GEQIMQJRPPRQZAMPMZSUJMAXW56MQ, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.028974090 (248h52m34.623085345s) [noise=true , address=145.79.227.33:4895 , piece_id=7IVIWLXB2TB3JPT6K553DDTKGYO3IDULZH6AD4LOQYQC7GZWYKGA, seg_pos={0 0}, upload=239266]: Uploading piece... Apr 18 01:09:24.181975315 (248h52m34.776086567s) [noise=true , address=167.17.72.70:4761 , piece_id=R4WKLF7LYTGV4QV43T22DE53TU7WOFMLIL64V254G3QPAOWMM3UQ, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:1zQX8ssQez6byb9qraaVviLHPN5n143VKNa5TY5kG64u8RB8Q4, address:167.17.72.70:4761): protocol: write tcp 139.59.144.22:37746->167.17.72.70:4761: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:09:24.544424220 (248h52m35.13853547s) [noise=true , address=145.79.227.21:4508 , piece_id=FRFW2WYAZY4XXYWUCYKVVXYE6C2P47PRNH2FNCTRO4VKF5FEHVAA, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:145.79.227.21:4508): protocol: storage node overloaded, request limit: 2000 Apr 18 01:09:24.607535558 (248h52m35.201646806s) [upload=239266]: Next returned nil segment Apr 18 01:09:24.607553332 (248h52m35.201664581s) [upload=239266]: Waiting for error group managing segments... Apr 18 01:09:24.744426162 (248h52m35.338537402s) [noise=true , address=145.79.227.18:4019 , piece_id=PFMRRQGGTIEFMDPUOIC45NJ6CBIQW3O6TSCCCKRYDNXZGLBU5IBA, seg_pos={0 0}, upload=239266]: Done uploading piece. err: <nil> Apr 18 01:09:25.996432141 (248h52m36.590543401s) [noise=true , address=64.49.13.140:4513 , piece_id=KPD55IU3GPNS7ASRYLLELY6TAPO5HMM22HA5XCADYFJQV6YPACIA, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:1Ac3N4hMJ66bTx2dpHBDT6W5SEGnPNfwyonwmvAANR8mM8vvpY, address:64.49.13.140:4513): protocol: storage node overloaded, request limit: 2000 Apr 18 01:09:26.118183013 (248h52m36.712294253s) [noise=true , address=77.90.31.18:4814 , piece_id=TMW4O7IBV5F6MTVG7TLW57446PVU44ZJSCXP2GQGDQSIVGBGP6RA, seg_pos={0 0}, upload=239266]: Done uploading piece. err: <nil> Apr 18 01:09:27.461890126 (248h52m38.056001365s) [noise=true , address=45.91.116.13:4180 , piece_id=5R2ZNYAJTRY4KGYH2SW67C4GEQIMQJRPPRQZAMPMZSUJMAXW56MQ, seg_pos={0 0}, upload=239266]: Done uploading piece. err: <nil> Apr 18 01:09:27.650146510 (248h52m38.24425775s) [noise=true , address=77.90.31.19:4860 , piece_id=HW4CFN2MN2DQVSW33MI7Z2PXFJ6TOO34CQNWH2ZIYKBCUP4C2CKQ, seg_pos={0 0}, upload=239266]: Done uploading piece. err: <nil> Apr 18 01:09:32.692340284 (248h52m43.286451524s) [noise=true , address=77.90.31.17:4792 , piece_id=73QIZUS4FT22OOQY2I5EAJZ5QGVM375M7K75AZYYTTXW6R2BQJHQ, seg_pos={0 0}, upload=239266]: Done uploading piece. err: <nil> Apr 18 01:09:35.955825904 (248h52m46.549937135s) [noise=true , address=145.79.227.9:4398 , piece_id=YUM6OTNKLIIYB6SIKPYEIQQ2DKKHCJKJI6GE6OPNXDYPMUHR4JXQ, seg_pos={0 0}, upload=239266]: Done uploading piece. err: <nil> Apr 18 01:10:23.609534291 (248h53m34.203645541s) [noise=true , address=145.79.227.34:4933 , piece_id=EJZJ7WUCVTNHB72JCSXDL7D3O7TFUYDJU5SBSZPME3KFRHTKCGCA, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:12Kj9KJZo29cpWQSprwxjWe82BrKnYYDTuKguPJm9hn8yVG83ou, address:145.79.227.34:4933): protocol: rpc: dial tcp 145.79.227.34:4933: i/o timeout; context deadline exceeded Apr 18 01:10:23.609764875 (248h53m34.203876124s) [noise=true , address=167.17.72.71:4794 , piece_id=4TB7C3COXSZFBLU5OTZYEGQ3JLMI3ZTIDFK67KVPRSXI4ZTH4AZA, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:1sNGPMY8nuWVLFECNta92W6sqAkSPBKDTMALh6ozm9ro9p94cY, address:167.17.72.71:4794): protocol: write tcp 139.59.144.22:38206->167.17.72.71:4794: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:10:23.610185863 (248h53m34.204297113s) [noise=true , address=145.79.227.29:4949 , piece_id=FS2FG4NBMFUUXGKJ5TPINN5PBEIGT4KBZLN7OSHQMLGEE23TNSHQ, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:12fLpEnaKeREB5itFjYy5yxy15a9GRT2w4p512qvB9wPTXZZwGo, address:145.79.227.29:4949): protocol: rpc: dial tcp 145.79.227.29:4949: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:10:23.610260715 (248h53m34.204371969s) [noise=true , address=193.149.164.143:4702 , piece_id=Z4DIMQQAAKIVYBU5A25UITI7KQO3XT6XNLUYV3RYMSBLGMVHUAEQ, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: write tcp 139.59.144.22:34016->193.149.164.143:4702: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:10:23.610525365 (248h53m34.204636614s) [noise=true , address=167.17.72.68:4647 , piece_id=ZTMQ2B65JW2FS3EZ5FINP7QSM6DAMTULMSEFFTUOGF6EGCJBWQMA, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:16EZPsM7oBshP77nG7JRSWUNjMbu56ut1VeQYeF9QDkcHXgbgt, address:167.17.72.68:4647): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:10:23.610820176 (248h53m34.204931425s) [noise=true , address=64.49.13.169:4706 , piece_id=MWUTUT6HQMBCKX7OCJMNADB3RSVOFR5AHTP6GQ4B4DDU5C7WQVAQ, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7, address:64.49.13.169:4706): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:10:23.610984503 (248h53m34.205095755s) [noise=true , address=145.79.227.11:4744 , piece_id=ZIRG5LCWV3VJWB23IDNI6XZOPID7O7R75JNMJ3R2AGSBFTER5MXA, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH, address:145.79.227.11:4744): protocol: read tcp 139.59.144.22:55052->145.79.227.11:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:10:23.611030258 (248h53m34.205141507s) [noise=true , address=167.17.72.69:4737 , piece_id=Q6HNZEMZDYPXGF2HWFENCTF24E2N6L4BYXNNWPWKX7OCV7E3O5HQ, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:12kzMj68hngmHwtxN8wEPDqP7dSQ729ztD357649iYViKCdSn34, address:167.17.72.69:4737): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:10:23.611160858 (248h53m34.205272109s) [noise=true , address=145.79.227.33:4895 , piece_id=7IVIWLXB2TB3JPT6K553DDTKGYO3IDULZH6AD4LOQYQC7GZWYKGA, seg_pos={0 0}, upload=239266]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:145.79.227.33:4895): protocol: rpc: dial tcp 145.79.227.33:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:10:23.611225770 (248h53m34.205337016s) [seg_pos={0 0}, upload=239266]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Apr 18 01:10:23.714268233 (248h53m34.308379474s) [upload=239266]: Done uploading segments... Apr 18 01:10:23.714325055 (248h53m34.308436297s) [upload=239266]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.107511043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXXX] duration: 5.0012177s trace: https://jaeger.dev.storj.io/trace/2f2f4e44fd25f181 FAIL (2026-04-18 01:10:00 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00128483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [.X....X..X..X..X....X....X.....X....XX..X..........XX...XX.X] duration: 1m0.001774122s trace: https://jaeger.dev.storj.io/trace/2547b4a4d41d6d7e FAIL (2026-04-18 01:09:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001802982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 42 out of 60 is failed (threshold=3) [.................XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.1100983s trace: https://jaeger.dev.storj.io/trace/2550b9d1b3ec3851 FAIL (2026-04-18 01:09:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 66:4995 , piece_id=4DJV5W66LMRRJL5KQXOACYWEVBJWCCFUVWXKBE2LGDH2KVIXSZEQ, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.586877133 (248h51m23.180988371s) [seg_pos={0 0}, upload=239249]: Done begin upload segment. Apr 18 01:08:12.586947089 (248h51m23.181058338s) [noise=true , address=45.91.116.14:4199 , piece_id=4CBZMLG4DTSAI5ODJXT2DXBCQUKKK5IJWMAPJOUTD4FAJNXEYCVQ, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.587454556 (248h51m23.181565801s) [noise=true , address=145.79.227.34:4914 , piece_id=5GYTB3SXIFFO7PVFLJXWGCSPVO24IKROOK2IYFJHPLV4VZ4THLSQ, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.587885794 (248h51m23.181997039s) [noise=true , address=193.149.164.71:4295 , piece_id=VKLVTHNOETAEGNIDKHBKH4QQIPVZCGHVDAY2GOYW6Q4YJXUQABFA, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.588393753 (248h51m23.182504999s) [noise=true , address=167.17.72.69:4730 , piece_id=AQVYIOYK6NVHYJZ4QZPPCE5RX6CPQ3NBHFNFH2RZB5N34IUTJRIQ, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.588841148 (248h51m23.182952401s) [noise=true , address=145.79.227.11:4739 , piece_id=Z6J3QX2ZGESNILSXER23P4GBVJSSIMZEMLWZNV4BB6FDOGCL3KAA, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.589420623 (248h51m23.183531867s) [noise=true , address=77.90.31.17:4800 , piece_id=5A3JW6BRCLDV4GEZKYBPIZSXIEBKQWYZTYSLD7MH3LIJDRBN42JA, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.590063795 (248h51m23.184175066s) [noise=true , address=145.79.227.49:4349 , piece_id=6GMMYIEOFPHAOESMMWJMQFK7YSCLGC6YDLV64R7ZEJ6T7MJ6QBQQ, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.590687239 (248h51m23.184798498s) [noise=true , address=145.79.227.18:4025 , piece_id=WACAGO57O7E5V4L3AHHKHA4XEZVCGVNDPF2HTF7AIS4XH5F32XFQ, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.591232600 (248h51m23.185343856s) [noise=true , address=77.90.31.12:4617 , piece_id=NJ2NWXN537G565C66HDLURUE52RZZNAKTO4BQZJDVD3T3E7K2HBQ, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.591785477 (248h51m23.185896723s) [noise=true , address=64.49.13.169:4706 , piece_id=7XDP2WMTZQYVIGGVCHCZMQCSAZQ7BMG6XB4YR7ZRKLXPMJ26HB5A, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.592968723 (248h51m23.187079971s) [noise=true , address=167.17.72.71:4811 , piece_id=YCSQART26ISNLXLXW6CMTKTGRF3PNOEW77GYDKCPTMNQMBLF5ZDQ, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.593347525 (248h51m23.187458781s) [noise=true , address=167.17.72.70:4754 , piece_id=PNXR4WWCAK6EE4ZKSF4ESCIBSY4QPXQZ23Z4666ZTZAJTMOTYSGA, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.595059238 (248h51m23.189170479s) [noise=true , address=145.79.227.13:4829 , piece_id=VY4QM6WPGQIOIQQOLCYDS5UXZCEAKYL3BOQRQGHUP337GQ7UU43Q, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.595118177 (248h51m23.189229434s) [noise=true , address=145.79.227.33:4897 , piece_id=IAQ7FVDAAO5PEPMX4WQKL6RZQSOOJLFAOUTZVGOVOMOGB4H7YQHA, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.596780203 (248h51m23.190891452s) [noise=true , address=145.79.227.9:4398 , piece_id=SZLV2EB4TYD4MS72HLGYMR3RMLAZAU5YTHSBDYRHZ5FUPMQU3ALA, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.596870046 (248h51m23.190981303s) [noise=true , address=167.17.72.68:4672 , piece_id=N2C74FKRJ35SGGUDZ27XYZMSFNXU7YZSPC73XTQ36M42VSRU75HQ, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.598729262 (248h51m23.192840504s) [noise=true , address=145.79.227.37:4572 , piece_id=TOSXU4V6NAWWU2XLA3DNAL6KIXWHBTIE4GOKSZKBIGBXLFSC4V6A, seg_pos={0 0}, upload=239249]: Uploading piece... Apr 18 01:08:12.687011368 (248h51m23.28112262s) [noise=true , address=145.79.227.49:4349 , piece_id=6GMMYIEOFPHAOESMMWJMQFK7YSCLGC6YDLV64R7ZEJ6T7MJ6QBQQ, seg_pos={0 0}, upload=239249]: Done uploading piece. err: ecclient: upload failed (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk, address:145.79.227.49:4349): protocol: rpc: dial tcp 145.79.227.49:4349: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:08:13.153714390 (248h51m23.747825624s) [upload=239249]: Next returned nil segment Apr 18 01:08:13.153734750 (248h51m23.747846007s) [upload=239249]: Waiting for error group managing segments... Apr 18 01:08:13.297241193 (248h51m23.891352436s) [noise=true , address=145.79.227.18:4025 , piece_id=WACAGO57O7E5V4L3AHHKHA4XEZVCGVNDPF2HTF7AIS4XH5F32XFQ, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:08:13.842137427 (248h51m24.436248666s) [noise=true , address=193.149.164.71:4295 , piece_id=VKLVTHNOETAEGNIDKHBKH4QQIPVZCGHVDAY2GOYW6Q4YJXUQABFA, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:08:13.874788553 (248h51m24.468899793s) [noise=true , address=193.149.164.66:4995 , piece_id=4DJV5W66LMRRJL5KQXOACYWEVBJWCCFUVWXKBE2LGDH2KVIXSZEQ, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:08:14.568267016 (248h51m25.162378255s) [noise=true , address=145.79.227.9:4398 , piece_id=SZLV2EB4TYD4MS72HLGYMR3RMLAZAU5YTHSBDYRHZ5FUPMQU3ALA, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:08:14.947066015 (248h51m25.541177254s) [noise=true , address=77.90.31.12:4617 , piece_id=NJ2NWXN537G565C66HDLURUE52RZZNAKTO4BQZJDVD3T3E7K2HBQ, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:08:17.262131775 (248h51m27.856243014s) [noise=true , address=45.91.116.14:4199 , piece_id=4CBZMLG4DTSAI5ODJXT2DXBCQUKKK5IJWMAPJOUTD4FAJNXEYCVQ, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:08:18.117828934 (248h51m28.711940186s) [noise=true , address=145.79.227.13:4829 , piece_id=VY4QM6WPGQIOIQQOLCYDS5UXZCEAKYL3BOQRQGHUP337GQ7UU43Q, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:08:20.645766362 (248h51m31.239877602s) [noise=true , address=77.90.31.17:4800 , piece_id=5A3JW6BRCLDV4GEZKYBPIZSXIEBKQWYZTYSLD7MH3LIJDRBN42JA, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:08:23.805565071 (248h51m34.399676311s) [noise=true , address=64.49.13.169:4706 , piece_id=7XDP2WMTZQYVIGGVCHCZMQCSAZQ7BMG6XB4YR7ZRKLXPMJ26HB5A, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:08:23.960536705 (248h51m34.554647952s) [noise=true , address=145.79.227.37:4572 , piece_id=TOSXU4V6NAWWU2XLA3DNAL6KIXWHBTIE4GOKSZKBIGBXLFSC4V6A, seg_pos={0 0}, upload=239249]: Done uploading piece. err: <nil> Apr 18 01:09:12.158653280 (248h52m22.752764527s) [noise=true , address=167.17.72.71:4811 , piece_id=YCSQART26ISNLXLXW6CMTKTGRF3PNOEW77GYDKCPTMNQMBLF5ZDQ, seg_pos={0 0}, upload=239249]: Done uploading piece. err: ecclient: upload failed (node:1gwAeSWHzY6H1YXbJ8WTkvpERNT9CR663Buc4cNUCJUpNjJMo9, address:167.17.72.71:4811): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:09:12.158719752 (248h52m22.752830993s) [noise=true , address=167.17.72.70:4754 , piece_id=PNXR4WWCAK6EE4ZKSF4ESCIBSY4QPXQZ23Z4666ZTZAJTMOTYSGA, seg_pos={0 0}, upload=239249]: Done uploading piece. err: ecclient: upload failed (node:12E93MNLYyGDZR1SuAN9Rdekxg9xRNCJQT1Gk6tSxStq8ck7CZJ, address:167.17.72.70:4754): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:09:12.159014438 (248h52m22.753125687s) [noise=true , address=167.17.72.69:4730 , piece_id=AQVYIOYK6NVHYJZ4QZPPCE5RX6CPQ3NBHFNFH2RZB5N34IUTJRIQ, seg_pos={0 0}, upload=239249]: Done uploading piece. err: ecclient: upload failed (node:125STJwNfwwMQ8V1nCHxFXrTBr9haCjGwdDVQhG67RwvgQKwaTr, address:167.17.72.69:4730): protocol: rpc: dial tcp 167.17.72.69:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:09:12.159052320 (248h52m22.753163569s) [noise=true , address=167.17.72.68:4672 , piece_id=N2C74FKRJ35SGGUDZ27XYZMSFNXU7YZSPC73XTQ36M42VSRU75HQ, seg_pos={0 0}, upload=239249]: Done uploading piece. err: ecclient: upload failed (node:12dtrjCvkcpwh4oTDjKe7Ate9JtzUw4BKrqpawATirSXUXkSp8r, address:167.17.72.68:4672): protocol: write tcp 139.59.144.22:45838->167.17.72.68:4672: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:09:12.159536739 (248h52m22.753647971s) [noise=true , address=145.79.227.33:4897 , piece_id=IAQ7FVDAAO5PEPMX4WQKL6RZQSOOJLFAOUTZVGOVOMOGB4H7YQHA, seg_pos={0 0}, upload=239249]: Done uploading piece. err: ecclient: upload failed (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU, address:145.79.227.33:4897): protocol: context deadline exceeded; context deadline exceeded Apr 18 01:09:12.159772681 (248h52m22.753883931s) [noise=true , address=145.79.227.11:4739 , piece_id=Z6J3QX2ZGESNILSXER23P4GBVJSSIMZEMLWZNV4BB6FDOGCL3KAA, seg_pos={0 0}, upload=239249]: Done uploading piece. err: ecclient: upload failed (node:1eEbhCXRGArodr2pffqXQxRR8w6u4c6J6auDbsCwe1NH5Wwycd, address:145.79.227.11:4739): protocol: rpc: dial tcp 145.79.227.11:4739: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:09:12.159854127 (248h52m22.753965378s) [noise=true , address=145.79.227.34:4914 , piece_id=5GYTB3SXIFFO7PVFLJXWGCSPVO24IKROOK2IYFJHPLV4VZ4THLSQ, seg_pos={0 0}, upload=239249]: Done uploading piece. err: ecclient: upload failed (node:12wWwD62SvgY72yFsUXtZMMvCea2xZkRtq72sB7aAkjHoyJWa6N, address:145.79.227.34:4914): protocol: rpc: dial tcp 145.79.227.34:4914: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:09:12.159892214 (248h52m22.754003446s) [seg_pos={0 0}, upload=239249]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Apr 18 01:09:12.266437954 (248h52m22.860549201s) [upload=239249]: Done uploading segments... Apr 18 01:09:12.266492160 (248h52m22.86060341s) [upload=239249]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.110123908s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXXX] duration: 5.001227866s trace: https://jaeger.dev.storj.io/trace/9c6bea1bd4ec9f7 FAIL (2026-04-18 01:09:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001253382s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXXX] duration: 5.00089111s trace: https://jaeger.dev.storj.io/trace/1379dc1d051faa37 FAIL (2026-04-18 01:08:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000918431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 41 out of 60 is failed (threshold=3) [..................XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.115978407s trace: https://jaeger.dev.storj.io/trace/625d50175d16a779 FAIL (2026-04-18 01:08:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== :07:12.299604523 (248h50m22.893715766s) [noise=true , address=145.79.227.40:4082 , piece_id=43HSKUE3Q6KUTOAGA6Q7YRDAAVRS4KUIRFO65GJHODOMU54GF4DQ, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.300035927 (248h50m22.894147179s) [noise=true , address=167.17.72.70:4766 , piece_id=75BZDUW4JM6KRDGHZMPY5Z6RH2YN5NRAH3SLNWIHG6ROFO7CDQKA, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.300696001 (248h50m22.894807248s) [noise=true , address=145.79.227.29:4968 , piece_id=L5R6GSC5PNUTLWV6REGHSSVRVWI47HR26QLF42TTEILCA5AB2S3A, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.301131394 (248h50m22.895242644s) [noise=true , address=77.90.31.17:4786 , piece_id=LN5MLSXBS45ABSAROO67V55JNV5LRY6TQR3TKTVIZL36K6SK6JLA, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.301557010 (248h50m22.895668257s) [noise=true , address=145.79.227.28:4735 , piece_id=5UZQ7WC26S44BKRZ6O4HZNMZSE6WAI7GZ7WGBDVHBXUWUZNMLIYQ, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.302026856 (248h50m22.896138108s) [noise=true , address=193.149.164.66:4986 , piece_id=HFDJBWZIOWLTX777YWHF3ODLY5ETSYCHR6UDAOG3T44XO3AUCGCQ, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.302520561 (248h50m22.896631811s) [noise=true , address=167.17.72.69:4715 , piece_id=7FWGTAFJNNSK5DQIBE7ICCJJTVQ2OM33HKGH26R6F5SOXTKCMZ2Q, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.303000414 (248h50m22.89711164s) [noise=true , address=145.79.227.11:4737 , piece_id=LD4QIFUMJ2L5RO6GEL55RNBRDFTNFIYQ2PXVYAV7XQ3CRALXJXSQ, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.305180300 (248h50m22.899291551s) [noise=true , address=145.79.227.13:4836 , piece_id=PEV5I2L3L2E3BPIJEAKXHOHIO233L4CJQIETF6OQ5Z5ANH7L7DAQ, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.306017211 (248h50m22.900128458s) [noise=true , address=145.79.227.44:4221 , piece_id=7GWTK4TF5LOCRLG7BM6S6XEMSMBEHCWM75IQ75PYFK7W73XGMUNQ, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.306264600 (248h50m22.900375896s) [noise=true , address=193.149.164.71:4290 , piece_id=QOEAJIBIWA5DBTNOKQGQWB42223XM3VKLBYZ6R5M7NEQFAHB4EZA, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.306766125 (248h50m22.900877373s) [noise=true , address=145.79.227.47:4293 , piece_id=IMK6JW7REZEFQSOQUBJ4M2RRCSKLYNPS6QH2LDYDMBECZSYQS3QQ, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.307449610 (248h50m22.901560859s) [noise=true , address=45.91.116.13:4180 , piece_id=5ZOZTQHKCAHO3UTO65ARYET6LJ6UASSJLKKOEZOANF5WNZRGGN2Q, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.308441171 (248h50m22.902552422s) [noise=true , address=167.17.72.68:4650 , piece_id=YSK3MFLQXU3WUT3Y3XYKI3GI5I7F2UXMXLYTZZN37ONRXI2DUCAQ, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.309477252 (248h50m22.903588503s) [noise=true , address=145.79.227.33:4895 , piece_id=WVPMAQYKOSVM35R2QGTJQ5IDVR2LLD2PDT6FRNM6X56XPXLBRSJQ, seg_pos={0 0}, upload=239235]: Uploading piece... Apr 18 01:07:12.443121822 (248h50m23.037233071s) [noise=true , address=167.17.72.68:4650 , piece_id=YSK3MFLQXU3WUT3Y3XYKI3GI5I7F2UXMXLYTZZN37ONRXI2DUCAQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:12VnVRKqd9wzV3r4DiR9KvbFPPxssAE2VudYYDFvgWtmAuVdGN6, address:167.17.72.68:4650): protocol: write tcp 139.59.144.22:52810->167.17.72.68:4650: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:07:13.012001444 (248h50m23.606112675s) [upload=239235]: Next returned nil segment Apr 18 01:07:13.012016105 (248h50m23.606127353s) [upload=239235]: Waiting for error group managing segments... Apr 18 01:07:13.322904560 (248h50m23.917015798s) [noise=true , address=45.91.116.13:4180 , piece_id=5ZOZTQHKCAHO3UTO65ARYET6LJ6UASSJLKKOEZOANF5WNZRGGN2Q, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: storage node overloaded, request limit: 2000 Apr 18 01:07:13.502049215 (248h50m24.096160458s) [noise=true , address=145.79.227.28:4735 , piece_id=5UZQ7WC26S44BKRZ6O4HZNMZSE6WAI7GZ7WGBDVHBXUWUZNMLIYQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: <nil> Apr 18 01:07:13.556267292 (248h50m24.150378533s) [noise=true , address=193.149.164.71:4290 , piece_id=QOEAJIBIWA5DBTNOKQGQWB42223XM3VKLBYZ6R5M7NEQFAHB4EZA, seg_pos={0 0}, upload=239235]: Done uploading piece. err: <nil> Apr 18 01:07:13.707681018 (248h50m24.301792259s) [noise=true , address=193.149.164.66:4986 , piece_id=HFDJBWZIOWLTX777YWHF3ODLY5ETSYCHR6UDAOG3T44XO3AUCGCQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: <nil> Apr 18 01:07:14.030690948 (248h50m24.624802187s) [noise=true , address=145.79.227.7:4326 , piece_id=DMWIIU6KFDAD4YLIWAANTY2H5BSHAYPAIJMIJRLHKBK6I7JZVA4A, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:12pU8MZZKS88CPFdZoaQ4msNfkCbCewTmS2wKEdN7pG5aCWoAr6, address:145.79.227.7:4326): protocol: storage node overloaded, request limit: 2000 Apr 18 01:07:17.326101536 (248h50m27.920212775s) [noise=true , address=145.79.227.47:4293 , piece_id=IMK6JW7REZEFQSOQUBJ4M2RRCSKLYNPS6QH2LDYDMBECZSYQS3QQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: <nil> Apr 18 01:07:21.847739304 (248h50m32.441850545s) [noise=true , address=77.90.31.17:4786 , piece_id=LN5MLSXBS45ABSAROO67V55JNV5LRY6TQR3TKTVIZL36K6SK6JLA, seg_pos={0 0}, upload=239235]: Done uploading piece. err: <nil> Apr 18 01:07:21.911581060 (248h50m32.505692305s) [noise=true , address=145.79.227.13:4836 , piece_id=PEV5I2L3L2E3BPIJEAKXHOHIO233L4CJQIETF6OQ5Z5ANH7L7DAQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: <nil> Apr 18 01:07:32.446232440 (248h50m43.040343685s) [noise=true , address=145.79.227.40:4082 , piece_id=43HSKUE3Q6KUTOAGA6Q7YRDAAVRS4KUIRFO65GJHODOMU54GF4DQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: <nil> Apr 18 01:08:11.865090258 (248h51m22.459201503s) [noise=true , address=145.79.227.34:4920 , piece_id=FTB3WSFLMVIPNH6MBFLDKGEXCKKFU6NYX7MSBPO4HVQYMAZ4OTTQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:12ryY6XP6e8yUpMgwUmwf5z5syrjj6jBqzMpt2AUL9yV8tbd18S, address:145.79.227.34:4920): protocol: rpc: dial tcp 145.79.227.34:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:08:11.865512367 (248h51m22.459623618s) [noise=true , address=145.79.227.11:4737 , piece_id=LD4QIFUMJ2L5RO6GEL55RNBRDFTNFIYQ2PXVYAV7XQ3CRALXJXSQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:145.79.227.11:4737): protocol: rpc: dial tcp 145.79.227.11:4737: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:08:11.865637882 (248h51m22.459749129s) [noise=true , address=145.79.227.44:4221 , piece_id=7GWTK4TF5LOCRLG7BM6S6XEMSMBEHCWM75IQ75PYFK7W73XGMUNQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:145.79.227.44:4221): protocol: read tcp 139.59.144.22:59830->145.79.227.44:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:08:11.865718372 (248h51m22.459829615s) [noise=true , address=193.149.164.143:4701 , piece_id=KRDJTCUE6X65PI35NPAN5IAGIGQO7E3T4LBV2GKXRRHCPXORJIZA, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:18gM2HhdK2fn7tQo7WowTWCBhjxF4t7RztaF8h9S5SccA4ALcz, address:193.149.164.143:4701): protocol: write tcp 139.59.144.22:60946->193.149.164.143:4701: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:08:11.866037270 (248h51m22.460148518s) [noise=true , address=167.17.72.70:4766 , piece_id=75BZDUW4JM6KRDGHZMPY5Z6RH2YN5NRAH3SLNWIHG6ROFO7CDQKA, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:19wX3zUggUB4E4BGkCWZ4aY4tUc2EbfVzb1wtw93C65r21WpHq, address:167.17.72.70:4766): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:08:11.866115421 (248h51m22.460226672s) [noise=true , address=167.17.72.69:4715 , piece_id=7FWGTAFJNNSK5DQIBE7ICCJJTVQ2OM33HKGH26R6F5SOXTKCMZ2Q, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:12bA2HZbfaYTXrufGaXHUdLXvN3J69fgtUKAivVqYaSkh6Hq2sr, address:167.17.72.69:4715): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:08:11.866367521 (248h51m22.46047877s) [noise=true , address=145.79.227.33:4895 , piece_id=WVPMAQYKOSVM35R2QGTJQ5IDVR2LLD2PDT6FRNM6X56XPXLBRSJQ, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:1p99EPp3sCPSTEivuuBYRv791ysLFqRBwBp1LCpuAmtpasydTu, address:145.79.227.33:4895): protocol: rpc: dial tcp 145.79.227.33:4895: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:08:11.866388883 (248h51m22.460500114s) [noise=true , address=145.79.227.29:4968 , piece_id=L5R6GSC5PNUTLWV6REGHSSVRVWI47HR26QLF42TTEILCA5AB2S3A, seg_pos={0 0}, upload=239235]: Done uploading piece. err: ecclient: upload failed (node:1J98jLqn1SVMYjTYxKnNqYbRxSQ5vaUnq2G7XpVngHTaWRJ8oX, address:145.79.227.29:4968): protocol: rpc: dial tcp 145.79.227.29:4968: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:08:11.866419688 (248h51m22.460530914s) [seg_pos={0 0}, upload=239235]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Apr 18 01:08:11.978349032 (248h51m22.572460278s) [upload=239235]: Done uploading segments... Apr 18 01:08:11.978391983 (248h51m22.57250323s) [upload=239235]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.115995595s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..X....X..X..X..X....X....X.....X....XX..X..........XX...XX.] duration: 14.347138845s trace: https://jaeger.dev.storj.io/trace/2978e8f5e595e56a FAIL (2026-04-18 01:07:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.347157564s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXXX] duration: 5.001122738s trace: https://jaeger.dev.storj.io/trace/1612adff55f424ea FAIL (2026-04-18 01:07:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001148217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...X....X..X..X..X....X....X.....X....XX..X..........XX...XX] duration: 1m0.001216401s trace: https://jaeger.dev.storj.io/trace/2978e8f5e595e56a FAIL (2026-04-18 01:07:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001235474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 40 out of 60 is failed (threshold=3) [...................XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.119170768s trace: https://jaeger.dev.storj.io/trace/7032ebcf2dd38976 FAIL (2026-04-18 01:07:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== IUOLMU65JMIBFV4DLTT7HR3NQRKMTRX2XQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.023350295 (248h49m22.617461541s) [noise=true , address=193.149.164.71:4312 , piece_id=6CQHFDL3JA5OHJDO3GA7BZ2KIUZIQ6YQA35ZAG6MBMN2DMJEOAEQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.023730039 (248h49m22.617841285s) [noise=true , address=193.149.164.66:4990 , piece_id=OPERUODMW3ELGN3WY7JBMGWLWI65M2YJR4KNBZGCWTZW24KYDSHQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.024112541 (248h49m22.618223781s) [noise=true , address=145.79.227.13:4813 , piece_id=EVHQQYFM3F7XXJAVUEZ3QTINVU7BECJVRJ4DJU6ON3XBWW2EF74A, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.024760535 (248h49m22.618871797s) [noise=true , address=45.91.116.13:4180 , piece_id=EHIEXY7PLRAAR4E3GAT4K5NP74IMDQHG74YXGLC6ZS3BTZIKKJTQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.025254732 (248h49m22.619365979s) [noise=true , address=145.79.227.40:4070 , piece_id=HXKPFE2BUVRYB7KS5D2EI7B3IDB7IRHAXTH33Z4VSLP27BBDZKZA, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.025735324 (248h49m22.619846564s) [noise=true , address=145.79.227.11:4748 , piece_id=YSUUED2NE5WH6DUS7Q5JDTB5Z6HF6ESSBYX27MKIUFCB4IRWXAAQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.028805848 (248h49m22.622917091s) [noise=true , address=145.79.227.2:4120 , piece_id=2YQ7AHSHKLBCFBOHUMXEBML2AHT6PNGTHSUOLD4YE3KP74EOUZWQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.030439287 (248h49m22.624550529s) [noise=true , address=167.17.72.71:4813 , piece_id=PMVBXZSL6MHPBS4FHM6E7P3KRIQYNUB6JH4VXCMU3X3KRDSKDZQA, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.031959131 (248h49m22.626070375s) [noise=true , address=145.79.227.29:4961 , piece_id=2VZ5MY4XP673QZUX26AVH5D3UZAD6L32OHTHLDCWWSD64HVNPXAQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.033503955 (248h49m22.627615207s) [noise=true , address=145.79.227.32:4840 , piece_id=7MDGRIOLZBLHNHL5HICJTD5WJEZN7DO2T7NFNK5VYBBQGSBTVSWA, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.035340417 (248h49m22.62945166s) [noise=true , address=145.79.227.41:4111 , piece_id=TMULBGMSPCYJWS2N3Z62RB737BGBHNUTIJ6H5XZBIV2UKAZJCRKQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.036945098 (248h49m22.631056341s) [noise=true , address=145.79.227.21:4508 , piece_id=YBM7MSQYQB7FMAU27QTGUK5Y3KI2XJH2VL2YBJFSB7PAN4ZI2TRA, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.038723723 (248h49m22.632834975s) [noise=true , address=167.17.72.68:4652 , piece_id=YQF7LTGGTHEUU7JDJDB6T4PABJWDEQADHDHHOKIFHEGPTO7WNBAQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.040282067 (248h49m22.634393316s) [noise=true , address=145.79.227.50:4398 , piece_id=YCTWTJLOVJ6YG2L3B7JHSJ6VIHOFJE2FLSJHJUSHAVKWYCTA2SAQ, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.040961280 (248h49m22.635072531s) [noise=true , address=167.17.72.69:4721 , piece_id=2MCEO4FPPWUJ6LQTQV6SXKSRW5QHJB3QESDXEL43Q3TYZDA476FA, seg_pos={0 0}, upload=239221]: Uploading piece... Apr 18 01:06:12.604640564 (248h49m23.198751823s) [upload=239221]: Next returned nil segment Apr 18 01:06:12.604661513 (248h49m23.19877277s) [upload=239221]: Waiting for error group managing segments... Apr 18 01:06:12.720957460 (248h49m23.315068705s) [noise=true , address=193.149.164.143:4700 , piece_id=TR5UEEVJP3CYWUUJBIIUOLMU65JMIBFV4DLTT7HR3NQRKMTRX2XQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:1tnTEYvqJ6PB2DsuXA28sEuEiGj11StA4qdhcYH1kuoaMkTVdY, address:193.149.164.143:4700): protocol: storage node overloaded, request limit: 2000 Apr 18 01:06:12.761293705 (248h49m23.355404953s) [noise=true , address=145.79.227.13:4813 , piece_id=EVHQQYFM3F7XXJAVUEZ3QTINVU7BECJVRJ4DJU6ON3XBWW2EF74A, seg_pos={0 0}, upload=239221]: Done uploading piece. err: <nil> Apr 18 01:06:12.788323307 (248h49m23.382434551s) [noise=true , address=145.79.227.32:4840 , piece_id=7MDGRIOLZBLHNHL5HICJTD5WJEZN7DO2T7NFNK5VYBBQGSBTVSWA, seg_pos={0 0}, upload=239221]: Done uploading piece. err: <nil> Apr 18 01:06:12.795071158 (248h49m23.389182404s) [noise=true , address=145.79.227.41:4111 , piece_id=TMULBGMSPCYJWS2N3Z62RB737BGBHNUTIJ6H5XZBIV2UKAZJCRKQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: <nil> Apr 18 01:06:13.290328483 (248h49m23.884439731s) [noise=true , address=193.149.164.71:4312 , piece_id=6CQHFDL3JA5OHJDO3GA7BZ2KIUZIQ6YQA35ZAG6MBMN2DMJEOAEQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: <nil> Apr 18 01:06:13.978350597 (248h49m24.572461839s) [noise=true , address=145.79.227.50:4398 , piece_id=YCTWTJLOVJ6YG2L3B7JHSJ6VIHOFJE2FLSJHJUSHAVKWYCTA2SAQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: <nil> Apr 18 01:06:14.083031233 (248h49m24.677142492s) [noise=true , address=45.91.116.11:4144 , piece_id=BXRXANKNCFJXQDCLMOPATUQL7KLL7DHK3WVW2V3BRCZ4HKBT4VPQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:12pVUbYqZ45XwL1okUwfeWoERLzHEv7Ne3dRMgmFvuxKXGktCiA, address:45.91.116.11:4144): protocol: storage node overloaded, request limit: 2000 Apr 18 01:06:15.469083436 (248h49m26.063194686s) [noise=true , address=45.91.116.13:4180 , piece_id=EHIEXY7PLRAAR4E3GAT4K5NP74IMDQHG74YXGLC6ZS3BTZIKKJTQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: <nil> Apr 18 01:06:15.893600013 (248h49m26.487711265s) [noise=true , address=145.79.227.21:4508 , piece_id=YBM7MSQYQB7FMAU27QTGUK5Y3KI2XJH2VL2YBJFSB7PAN4ZI2TRA, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:145.79.227.21:4508): protocol: storage node overloaded, request limit: 2000 Apr 18 01:06:24.480027238 (248h49m35.074138477s) [noise=true , address=145.79.227.2:4120 , piece_id=2YQ7AHSHKLBCFBOHUMXEBML2AHT6PNGTHSUOLD4YE3KP74EOUZWQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: <nil> Apr 18 01:07:11.593138664 (248h50m22.187249917s) [noise=true , address=145.79.227.11:4748 , piece_id=YSUUED2NE5WH6DUS7Q5JDTB5Z6HF6ESSBYX27MKIUFCB4IRWXAAQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:145.79.227.11:4748): protocol: rpc: dial tcp 145.79.227.11:4748: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:07:11.593443410 (248h50m22.18755466s) [noise=true , address=193.149.164.66:4990 , piece_id=OPERUODMW3ELGN3WY7JBMGWLWI65M2YJR4KNBZGCWTZW24KYDSHQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:12rZoTqAdgDDLWrSLga3M8CKSCNayUX33YFx5HB6g1FDjrVHUcq, address:193.149.164.66:4990): protocol: write tcp 139.59.144.22:37496->193.149.164.66:4990: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:07:11.593780961 (248h50m22.187892212s) [noise=true , address=145.79.227.33:4889 , piece_id=KPFBHDMYILZV3POFFIU5RRIK5E5PO6JUF2AMVSKDARLVP42SMYWQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:1PCtJRFHPHi1dHJNcRnGZg8HZPQDfo5L3aw6DgVnUzVUz92Kyv, address:145.79.227.33:4889): protocol: rpc: dial tcp 145.79.227.33:4889: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:07:11.594095031 (248h50m22.188206281s) [noise=true , address=167.17.72.68:4652 , piece_id=YQF7LTGGTHEUU7JDJDB6T4PABJWDEQADHDHHOKIFHEGPTO7WNBAQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:12mJEqAax1wUE4FaQEmCkHM5DqgUZxvR8e9WFYZTY4ohqLzinrr, address:167.17.72.68:4652): protocol: rpc: dial tcp 167.17.72.68:4652: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:07:11.594125450 (248h50m22.1882367s) [noise=true , address=167.17.72.71:4813 , piece_id=PMVBXZSL6MHPBS4FHM6E7P3KRIQYNUB6JH4VXCMU3X3KRDSKDZQA, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:12ibPgBMP6M4S4cxMwHmAFAtnGNvhXR9SQqfBiQm9sWE9LowA42, address:167.17.72.71:4813): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:07:11.594480319 (248h50m22.188591573s) [noise=true , address=145.79.227.29:4961 , piece_id=2VZ5MY4XP673QZUX26AVH5D3UZAD6L32OHTHLDCWWSD64HVNPXAQ, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:12ZGPLhM6NsdjKLkLCCLSeaNS8ikfnHPZxc3CCw9mLzSznbEoUa, address:145.79.227.29:4961): protocol: rpc: dial tcp 145.79.227.29:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:07:11.594694034 (248h50m22.188805284s) [noise=true , address=167.17.72.69:4721 , piece_id=2MCEO4FPPWUJ6LQTQV6SXKSRW5QHJB3QESDXEL43Q3TYZDA476FA, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:1gWFQbHXFFuQMNRRpTqpM6V2R9DJmPtogsxDkgWES3pGhBddyN, address:167.17.72.69:4721): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:07:11.594937978 (248h50m22.18904923s) [noise=true , address=145.79.227.40:4070 , piece_id=HXKPFE2BUVRYB7KS5D2EI7B3IDB7IRHAXTH33Z4VSLP27BBDZKZA, seg_pos={0 0}, upload=239221]: Done uploading piece. err: ecclient: upload failed (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me, address:145.79.227.40:4070): protocol: read tcp 139.59.144.22:37346->145.79.227.40:4070: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:07:11.594974267 (248h50m22.189085498s) [seg_pos={0 0}, upload=239221]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Apr 18 01:07:11.709027696 (248h50m22.303138937s) [upload=239221]: Done uploading segments... Apr 18 01:07:11.709065184 (248h50m22.303176425s) [upload=239221]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.119194099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXXX] duration: 5.00169928s trace: https://jaeger.dev.storj.io/trace/191931704c90fee8 FAIL (2026-04-18 01:06:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001738426s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [....X....X..X..X..X....X....X.....X....XX..X..........XX...X] duration: 1m0.001425963s trace: https://jaeger.dev.storj.io/trace/14d1a6dd7f053e46 FAIL (2026-04-18 01:06:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001450002s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXXX] duration: 5.000956995s trace: https://jaeger.dev.storj.io/trace/540cefa4bba60d34 FAIL (2026-04-18 01:06:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000983956s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 39 out of 60 is failed (threshold=3) [....................XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.111049131s trace: https://jaeger.dev.storj.io/trace/5f18919a9167f034 FAIL (2026-04-18 01:05:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== 57.056477340 (248h48m7.650588597s) [noise=true , address=145.79.227.32:4864 , piece_id=GQSJGUIYD4UPWGCDPSNML7TVKMSCDID3NONEV5UXNNFDCLKJSHCQ, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.057130809 (248h48m7.651242065s) [noise=true , address=145.79.227.11:4752 , piece_id=J2HSXZDIMXJJCAA6AEWT6VHMSIDIGESXFDAHO7EZVM5VLZIQMKQA, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.057351488 (248h48m7.651462738s) [noise=true , address=145.79.227.18:4026 , piece_id=Q6UUSFXUJID2RV5ACKP4PYTOQ7KBSLFD3IR6TR3HRWZQSFILJDMQ, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.057883289 (248h48m7.651994538s) [noise=true , address=193.149.164.71:4300 , piece_id=ZLN3KW5IB3TFWWWIU36NDBZ4FS3ON5TIXI45T3AX6IMMSNFR65IQ, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.058298888 (248h48m7.652410132s) [noise=true , address=145.79.227.13:4809 , piece_id=XADY7L4QY5PR2AUZFD44527CV2FHTEONK4KXITTJRG574BDOWEKA, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.058392330 (248h48m7.652503576s) [noise=true , address=145.79.227.2:4120 , piece_id=YGNYPMS3KOGV2ZHH5PO7LA7UY5ZGHW7AI2UK5SKAGJB7O6K666MQ, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.058676616 (248h48m7.652787867s) [noise=true , address=145.79.227.49:4372 , piece_id=6TF7JNMC3W4ZGY7FALXWY6CPME44T2UWKVMRQUGO73YCTO3DMEVA, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.058852067 (248h48m7.65296332s) [noise=true , address=167.17.72.71:4789 , piece_id=37LOULFDOTF7RM5LBSLCSS33QJKUJM22RJ6FFXLKKYCYT2KDKA4Q, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.059228900 (248h48m7.653340152s) [noise=true , address=145.79.227.43:4191 , piece_id=VHXO42CZ5TD4236MR6THAHXJDRYYNPO2LHSFEMAXWHGL42QY7LPQ, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.059288232 (248h48m7.653399473s) [noise=true , address=145.79.227.38:4005 , piece_id=BJUTZ6N5ERAINHY2MV3IUUTUSJT4MZRD2UFI73NEFMAKEUYO53MQ, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.059640706 (248h48m7.653751948s) [noise=true , address=145.79.227.34:4928 , piece_id=CX2BTP32QERITLUOTCQWJ4V62ATJBOYABTJEYO73FYSZHKK7JTAA, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.059745636 (248h48m7.653856883s) [noise=true , address=167.17.72.68:4644 , piece_id=22LQ72BJWQHVNO233UT22CUQZGLRSBEG76NMABV5WSJ3RCB6SZ2A, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.059977783 (248h48m7.654089022s) [noise=true , address=167.17.72.70:4775 , piece_id=YHM6YSF7RA44UR44ID7KJXXFZHTDWB4F2V2L4GGVRYDST7NL4RXA, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.060472824 (248h48m7.654584071s) [noise=true , address=145.79.227.31:4823 , piece_id=2FJKGLNSNGYFUWTZXLTK5U44P3FLYHQG7XMNG2UJQO6XWRXLZHLQ, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.060573744 (248h48m7.654685s) [noise=true , address=193.149.164.66:4986 , piece_id=5334PXBHNLWYL3HHU5YLT7E45GJ6DKHOVLNR3MOCM6IETYNZHHKQ, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.060953795 (248h48m7.655065037s) [noise=true , address=167.17.72.69:4709 , piece_id=SBV2YEC6WVC33S3Q2QGU6POQR24ENZUBWCFUTOS6HIPNIOX4XR6Q, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.065108773 (248h48m7.659220023s) [noise=true , address=64.49.13.138:4483 , piece_id=QZSY5UQ7P3LJR5RDABEHFGHEK27DPBR4ALOMO3QPPSAQG5ZME2YA, seg_pos={0 0}, upload=239199]: Uploading piece... Apr 18 01:04:57.149651960 (248h48m7.743763202s) [noise=true , address=145.79.227.49:4372 , piece_id=6TF7JNMC3W4ZGY7FALXWY6CPME44T2UWKVMRQUGO73YCTO3DMEVA, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:1kJnsSPzFa7JwUMHuvosMmFrGvQ8tCsQbjwWdxbqjHNHgTfaL3, address:145.79.227.49:4372): protocol: rpc: dial tcp 145.79.227.49:4372: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:04:57.625141176 (248h48m8.219252416s) [upload=239199]: Next returned nil segment Apr 18 01:04:57.625154743 (248h48m8.219265986s) [upload=239199]: Waiting for error group managing segments... Apr 18 01:04:57.655387076 (248h48m8.249498327s) [noise=true , address=167.17.72.69:4709 , piece_id=SBV2YEC6WVC33S3Q2QGU6POQR24ENZUBWCFUTOS6HIPNIOX4XR6Q, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:127QjpPepBunapakyCuqDQyXz4yJ4GaKcHA2meAEsCfY8Jkgkb8, address:167.17.72.69:4709): protocol: storage node overloaded, request limit: 2000 Apr 18 01:04:57.772878917 (248h48m8.366990168s) [noise=true , address=145.79.227.18:4026 , piece_id=Q6UUSFXUJID2RV5ACKP4PYTOQ7KBSLFD3IR6TR3HRWZQSFILJDMQ, seg_pos={0 0}, upload=239199]: Done uploading piece. err: <nil> Apr 18 01:04:57.807979402 (248h48m8.402090644s) [noise=true , address=145.79.227.13:4809 , piece_id=XADY7L4QY5PR2AUZFD44527CV2FHTEONK4KXITTJRG574BDOWEKA, seg_pos={0 0}, upload=239199]: Done uploading piece. err: <nil> Apr 18 01:04:57.853691252 (248h48m8.447802494s) [noise=true , address=145.79.227.32:4864 , piece_id=GQSJGUIYD4UPWGCDPSNML7TVKMSCDID3NONEV5UXNNFDCLKJSHCQ, seg_pos={0 0}, upload=239199]: Done uploading piece. err: <nil> Apr 18 01:04:58.347993549 (248h48m8.942104791s) [noise=true , address=193.149.164.66:4986 , piece_id=5334PXBHNLWYL3HHU5YLT7E45GJ6DKHOVLNR3MOCM6IETYNZHHKQ, seg_pos={0 0}, upload=239199]: Done uploading piece. err: <nil> Apr 18 01:04:58.369248658 (248h48m8.963359905s) [noise=true , address=64.49.13.138:4483 , piece_id=QZSY5UQ7P3LJR5RDABEHFGHEK27DPBR4ALOMO3QPPSAQG5ZME2YA, seg_pos={0 0}, upload=239199]: Done uploading piece. err: <nil> Apr 18 01:04:58.476067819 (248h48m9.070179062s) [noise=true , address=193.149.164.71:4300 , piece_id=ZLN3KW5IB3TFWWWIU36NDBZ4FS3ON5TIXI45T3AX6IMMSNFR65IQ, seg_pos={0 0}, upload=239199]: Done uploading piece. err: <nil> Apr 18 01:05:04.367969778 (248h48m14.96208103s) [noise=true , address=145.79.227.31:4823 , piece_id=2FJKGLNSNGYFUWTZXLTK5U44P3FLYHQG7XMNG2UJQO6XWRXLZHLQ, seg_pos={0 0}, upload=239199]: Done uploading piece. err: <nil> Apr 18 01:05:10.113658358 (248h48m20.707769607s) [noise=true , address=145.79.227.2:4120 , piece_id=YGNYPMS3KOGV2ZHH5PO7LA7UY5ZGHW7AI2UK5SKAGJB7O6K666MQ, seg_pos={0 0}, upload=239199]: Done uploading piece. err: <nil> Apr 18 01:05:24.286315272 (248h48m34.880426511s) [noise=true , address=145.79.227.43:4191 , piece_id=VHXO42CZ5TD4236MR6THAHXJDRYYNPO2LHSFEMAXWHGL42QY7LPQ, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:145.79.227.43:4191): protocol: storage node overloaded, request limit: 2000 Apr 18 01:05:56.035661552 (248h49m6.629772791s) [noise=true , address=145.79.227.40:4064 , piece_id=QN6NMCYY3C3AITFTDQ4STRQTNQEKIIU4CXZVDONOCQCL5OEHBGOQ, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:1HtP28UAricpA5ZBZrn2eLK1GBQxFGL2UNchDUFViqMhw4pXjy, address:145.79.227.40:4064): protocol: read tcp 139.59.144.22:40204->145.79.227.40:4064: read: connection reset by peer; manager closed: conn closed: op Apr 18 01:05:56.639374632 (248h49m7.233485878s) [noise=true , address=167.17.72.70:4775 , piece_id=YHM6YSF7RA44UR44ID7KJXXFZHTDWB4F2V2L4GGVRYDST7NL4RXA, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:123bCtKLV9XfWqWDL1keVQQXaWnMk1Yvu6PghnvpxUsFwt2zCRg, address:167.17.72.70:4775): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:05:56.639548571 (248h49m7.233659819s) [noise=true , address=145.79.227.34:4928 , piece_id=CX2BTP32QERITLUOTCQWJ4V62ATJBOYABTJEYO73FYSZHKK7JTAA, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:12B6ymfqNvqiDsAbkF8H4ExxEuYaYb7gYQ45Y5GF4PGAAwmjbyd, address:145.79.227.34:4928): protocol: rpc: dial tcp 145.79.227.34:4928: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:05:56.639558804 (248h49m7.23367005s) [noise=true , address=167.17.72.71:4789 , piece_id=37LOULFDOTF7RM5LBSLCSS33QJKUJM22RJ6FFXLKKYCYT2KDKA4Q, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:12AcmR5QqJktD68Ae2jycDEqL13se2yhT42JoVgDWMQCPCPbf9e, address:167.17.72.71:4789): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:05:56.639737779 (248h49m7.233849026s) [noise=true , address=167.17.72.68:4644 , piece_id=22LQ72BJWQHVNO233UT22CUQZGLRSBEG76NMABV5WSJ3RCB6SZ2A, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:1rLSShF2FHDbmPpR6U3xMpQRaSDcraVeCqSHBcETu9jvSEgzf6, address:167.17.72.68:4644): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:05:56.639836689 (248h49m7.233947942s) [noise=true , address=145.79.227.11:4752 , piece_id=J2HSXZDIMXJJCAA6AEWT6VHMSIDIGESXFDAHO7EZVM5VLZIQMKQA, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:12oRYWQwoe5JtDCqfSShqtyRWVQLjL4UtbokBgNpDAocBKv1vqL, address:145.79.227.11:4752): protocol: rpc: dial tcp 145.79.227.11:4752: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:05:56.640017809 (248h49m7.234129057s) [noise=true , address=145.79.227.38:4005 , piece_id=BJUTZ6N5ERAINHY2MV3IUUTUSJT4MZRD2UFI73NEFMAKEUYO53MQ, seg_pos={0 0}, upload=239199]: Done uploading piece. err: ecclient: upload failed (node:1AivThSVLAp9zneNK79sic4AqnTQ62yvNTFF36imigarMGqDSW, address:145.79.227.38:4005): protocol: rpc: dial tcp 145.79.227.38:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:05:56.640061106 (248h49m7.234172331s) [seg_pos={0 0}, upload=239199]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Apr 18 01:05:56.748439927 (248h49m7.342551176s) [upload=239199]: Done uploading segments... Apr 18 01:05:56.748488672 (248h49m7.342599936s) [upload=239199]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.11106615s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.....X....X..X..X..X....X....X.....X....XX..X..........XX...] duration: 24.974376453s trace: https://jaeger.dev.storj.io/trace/76d1b8f34010c80b FAIL (2026-04-18 01:05:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.974403061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 38 out of 60 is failed (threshold=3) [.....................XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.112816187s trace: https://jaeger.dev.storj.io/trace/4b1517ab6010872 FAIL (2026-04-18 01:04:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== .91.116.11:4128 , piece_id=EASLGLNRU5U4PSL3X6EDRJWEM5GYI2TUHJXAS47A2RJWABRD36VQ, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.750100539 (248h47m7.344211783s) [noise=true , address=145.79.227.37:4596 , piece_id=FUTN6QXWD3YEVHSGQ3KSROCZQ6XWQ2XINVGI2DNJFJ7Y5XJBDWSQ, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.750533051 (248h47m7.344644306s) [noise=true , address=167.17.72.71:4816 , piece_id=CC6P4KWAVYT6IP5OIANAGIQKDSKS7FNIGUNCODZACFVCEOSVKO6A, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.750714141 (248h47m7.344825389s) [noise=true , address=145.79.227.44:4200 , piece_id=3TSUGON2FDGYVKZDY2TI4YWQQAKEKZAOCYKP35YEEIQGMJHLVEWA, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.751096403 (248h47m7.345207655s) [noise=true , address=77.90.31.18:4814 , piece_id=E64E7D5NMQVH2JK2ALV4QBWZKLKUFMASDNY3HNLGT7ECYWU75FNA, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.751185098 (248h47m7.34529635s) [noise=true , address=167.17.72.70:4756 , piece_id=JYN5COQBDAK356JPKXNGA4PFABQP4U3DTJDCUTK42PSIW7GFXMUQ, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.751701932 (248h47m7.345813185s) [noise=true , address=193.149.164.143:4694 , piece_id=G4M2U73EKCJFDNLXJB7JH3BHOCSFQWP7GNDZKUZCXOXD6PT7I4LQ, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.752621928 (248h47m7.346733179s) [noise=true , address=77.90.31.17:4778 , piece_id=JS6GLZHGKRZDSV3HGXYKFYCIS2EWSQCJ6JF267MJIUEY4VEJO23A, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.754355958 (248h47m7.348467258s) [noise=true , address=145.79.227.6:4273 , piece_id=IHP6WJLCWKBZIDV6UZL5TT56DCTF37IBH7ZMIFBRTHJFHIHG26KA, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.756111267 (248h47m7.350222512s) [noise=true , address=145.79.227.31:4831 , piece_id=SZFRIOK427WX5D37IF4SCZJTUCZSBI7SH7NODFZO3PIEGIJPJHLA, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.756950299 (248h47m7.351061543s) [noise=true , address=145.79.227.11:4761 , piece_id=CIL2NZAFL6DX7NGGS56JAPZHCQHJDMM2WDHJWDZKRBXNWZPB7PTQ, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.758356418 (248h47m7.352467664s) [noise=true , address=167.17.72.68:4666 , piece_id=VPYGKQYBUKNZJXSVMBTUFDIVNGMBW4XJD2WFV35NYKUHUVXHCK6A, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.761335199 (248h47m7.355446445s) [noise=true , address=145.79.227.43:4189 , piece_id=NVKOUF7GJ23HXE7P5GHTYH44LQQS42FFM66RVKZ2A5NUMGQEZ2QA, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.763129110 (248h47m7.357240354s) [noise=true , address=145.79.227.38:4998 , piece_id=OOTO67FWVNKWZT2Z56RFSNKCATOPT5ZT2UZZCM2S66O6CDYHFSOQ, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.763949719 (248h47m7.35806096s) [noise=true , address=145.79.227.33:4891 , piece_id=JQWMKKLV4KDRWRCB7QQAEE4ACA3G4HJ5PP3TVWUR3RW2KCONDOSA, seg_pos={0 0}, upload=239184]: Uploading piece... Apr 18 01:03:56.900953338 (248h47m7.495064587s) [noise=true , address=167.17.72.68:4666 , piece_id=VPYGKQYBUKNZJXSVMBTUFDIVNGMBW4XJD2WFV35NYKUHUVXHCK6A, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:121qq3B9v2xUYb5yy1BrCrA1CC3EwZkD3hZH4GJmwV68t9bswC3, address:167.17.72.68:4666): protocol: write tcp 139.59.144.22:56500->167.17.72.68:4666: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:03:57.336022730 (248h47m7.930133966s) [upload=239184]: Next returned nil segment Apr 18 01:03:57.336041545 (248h47m7.930152878s) [upload=239184]: Waiting for error group managing segments... Apr 18 01:03:57.513582213 (248h47m8.107693452s) [noise=true , address=145.79.227.13:4813 , piece_id=YMTCZB75S53KP66TPDJ2OW5EDPSOIKKVOJAYRAFRTSXMVKCCUWHQ, seg_pos={0 0}, upload=239184]: Done uploading piece. err: <nil> Apr 18 01:03:58.063553767 (248h47m8.657665021s) [noise=true , address=193.149.164.66:4970 , piece_id=76PO7VW6RD4W7SNHFXDTTYZKCMSLYBRH6TI34W4W2HPQMUTIELSA, seg_pos={0 0}, upload=239184]: Done uploading piece. err: <nil> Apr 18 01:03:58.149312184 (248h47m8.743423423s) [noise=true , address=193.149.164.71:4317 , piece_id=A4LYUUH3PQQ43KQESGZZI4JQQ5V5YGJ5QZO242SBQAD7W2JRRJTA, seg_pos={0 0}, upload=239184]: Done uploading piece. err: <nil> Apr 18 01:03:58.973463963 (248h47m9.567575203s) [noise=true , address=145.79.227.6:4273 , piece_id=IHP6WJLCWKBZIDV6UZL5TT56DCTF37IBH7ZMIFBRTHJFHIHG26KA, seg_pos={0 0}, upload=239184]: Done uploading piece. err: <nil> Apr 18 01:03:59.761861142 (248h47m10.355972382s) [noise=true , address=45.91.116.11:4128 , piece_id=EASLGLNRU5U4PSL3X6EDRJWEM5GYI2TUHJXAS47A2RJWABRD36VQ, seg_pos={0 0}, upload=239184]: Done uploading piece. err: <nil> Apr 18 01:04:00.859171558 (248h47m11.45328281s) [noise=true , address=77.90.31.18:4814 , piece_id=E64E7D5NMQVH2JK2ALV4QBWZKLKUFMASDNY3HNLGT7ECYWU75FNA, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf, address:77.90.31.18:4814): protocol: storage node overloaded, request limit: 2000 Apr 18 01:04:02.899805549 (248h47m13.493916801s) [noise=true , address=145.79.227.37:4596 , piece_id=FUTN6QXWD3YEVHSGQ3KSROCZQ6XWQ2XINVGI2DNJFJ7Y5XJBDWSQ, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:12uVgAqShxPQefEKUAZELdhDztRYA5Xqi8BjT3gwpa5QiEHk8Au, address:145.79.227.37:4596): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s1/f9/log-000000000001aaf9-00000000: read-only file system; EOF Apr 18 01:04:05.396916468 (248h47m15.991027718s) [noise=true , address=77.90.31.17:4778 , piece_id=JS6GLZHGKRZDSV3HGXYKFYCIS2EWSQCJ6JF267MJIUEY4VEJO23A, seg_pos={0 0}, upload=239184]: Done uploading piece. err: <nil> Apr 18 01:04:13.702676604 (248h47m24.296787845s) [noise=true , address=193.149.164.143:4694 , piece_id=G4M2U73EKCJFDNLXJB7JH3BHOCSFQWP7GNDZKUZCXOXD6PT7I4LQ, seg_pos={0 0}, upload=239184]: Done uploading piece. err: <nil> Apr 18 01:04:56.357585626 (248h48m6.951696876s) [noise=true , address=145.79.227.44:4200 , piece_id=3TSUGON2FDGYVKZDY2TI4YWQQAKEKZAOCYKP35YEEIQGMJHLVEWA, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:145.79.227.44:4200): protocol: rpc: dial tcp 145.79.227.44:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:04:56.357771746 (248h48m6.951882996s) [noise=true , address=167.17.72.71:4816 , piece_id=CC6P4KWAVYT6IP5OIANAGIQKDSKS7FNIGUNCODZACFVCEOSVKO6A, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:122Mtq4k2hDVC7GVp5vfSpHqnQuhW9WSy9Ry6XAW2Xgt32JyJ9b, address:167.17.72.71:4816): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:04:56.358035415 (248h48m6.952146667s) [noise=true , address=167.17.72.70:4756 , piece_id=JYN5COQBDAK356JPKXNGA4PFABQP4U3DTJDCUTK42PSIW7GFXMUQ, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:129RnCRryqH2vqWv6sSeTz12Xh5tvinPLHHL4pavvaxUbyHRGiF, address:167.17.72.70:4756): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:04:56.358169802 (248h48m6.952281055s) [noise=true , address=145.79.227.38:4998 , piece_id=OOTO67FWVNKWZT2Z56RFSNKCATOPT5ZT2UZZCM2S66O6CDYHFSOQ, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:1LV3A9EyGYTRvni2ENn6yYkBdxQiJFW6zLC9vikrezvz6F9dEX, address:145.79.227.38:4998): protocol: rpc: dial tcp 145.79.227.38:4998: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:04:56.358396713 (248h48m6.952507964s) [noise=true , address=145.79.227.33:4891 , piece_id=JQWMKKLV4KDRWRCB7QQAEE4ACA3G4HJ5PP3TVWUR3RW2KCONDOSA, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:145.79.227.33:4891): protocol: rpc: dial tcp 145.79.227.33:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:04:56.358529744 (248h48m6.952641011s) [noise=true , address=145.79.227.31:4831 , piece_id=SZFRIOK427WX5D37IF4SCZJTUCZSBI7SH7NODFZO3PIEGIJPJHLA, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:145.79.227.31:4831): protocol: rpc: dial tcp 145.79.227.31:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:04:56.358653422 (248h48m6.952764675s) [noise=true , address=145.79.227.11:4761 , piece_id=CIL2NZAFL6DX7NGGS56JAPZHCQHJDMM2WDHJWDZKRBXNWZPB7PTQ, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:12CXHkaBwi15pBUoRiZHi6JP5rBDu1iVpWT1WFL3dCmk8LRRxqw, address:145.79.227.11:4761): protocol: rpc: dial tcp 145.79.227.11:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:04:56.358879843 (248h48m6.952991087s) [noise=true , address=145.79.227.43:4189 , piece_id=NVKOUF7GJ23HXE7P5GHTYH44LQQS42FFM66RVKZ2A5NUMGQEZ2QA, seg_pos={0 0}, upload=239184]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:145.79.227.43:4189): protocol: read tcp 139.59.144.22:54876->145.79.227.43:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:04:56.358909910 (248h48m6.953021143s) [seg_pos={0 0}, upload=239184]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Apr 18 01:04:56.467528767 (248h48m7.061640018s) [upload=239184]: Done uploading segments... Apr 18 01:04:56.467569962 (248h48m7.06168121s) [upload=239184]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.112838147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XXX] duration: 5.001727239s trace: https://jaeger.dev.storj.io/trace/2433c7635c387cfc FAIL (2026-04-18 01:04:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001755702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 37 out of 60 is failed (threshold=3) [......................XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.111065096s trace: https://jaeger.dev.storj.io/trace/bbbef5c5f0c643b FAIL (2026-04-18 01:03:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== EUTR2ONG6TK64NSCSHOCGN5EYKF7YWGQ, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.507912083 (248h46m7.102023331s) [noise=true , address=145.79.227.31:4816 , piece_id=AD66F4MD2ZWBVF7BAJEE3TZV63ZSXNIAV3NSDI3KNHN53O6N4MJA, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.508112852 (248h46m7.102224104s) [noise=true , address=145.79.227.13:4836 , piece_id=6VLRGLPLLMHFCJL7SBCA3K6CRW2GLM2L52NAIS32772OQAIZGAAA, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.508516404 (248h46m7.10262766s) [noise=true , address=145.79.227.32:4841 , piece_id=KLVYZYDXDVLNQXV7K2C3NGTYJ7DUZ5MNFWHETCMGXKHU7XXMPF6A, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.508874464 (248h46m7.102985714s) [noise=true , address=167.17.72.71:4803 , piece_id=TFTSTBRC6DXP6XFSATU6IPXVYXBKMI3Z276XNOGCRAP6PSKK2RAA, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.509081564 (248h46m7.103192819s) [noise=true , address=145.79.227.29:4962 , piece_id=V44ZJREUTIQ6Y44SPIBYZKDDHJOPBWYG7TAKVSIDMJGLHLAOH2PA, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.509436747 (248h46m7.103547989s) [noise=true , address=145.79.227.35:4496 , piece_id=UFHQUD5KS6SP5TDMSNKRBHY6UURWLIS5PRB2FM4NZ72ZFAKXRC5A, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.509621435 (248h46m7.103732687s) [noise=true , address=167.17.72.70:4772 , piece_id=M5LFDCH57OKMYEYSMTZLBMMON6AHVMTXNLDDLFKRHKJ2FEW4HZ6Q, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.509966995 (248h46m7.104078238s) [noise=true , address=145.79.227.38:4010 , piece_id=5WRR2YTDS5ERLD36KBHLSZBVM6YPBOPFOHLRQV4XTDRWN3K7FZQQ, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.510522714 (248h46m7.104633958s) [noise=true , address=167.17.72.69:4744 , piece_id=J2BH2OXEQBRJE4ZTHGWM3IGQHXWXDSJTS7XNBL5PVYXWTLF2XIWA, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.511135145 (248h46m7.105246453s) [noise=true , address=145.79.227.44:4202 , piece_id=JU3VAO6347F2ID2ZXRHN6N3PQQ7VGOFFTGAWPUHOMEFTCNCQQG5A, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.511739261 (248h46m7.105850502s) [noise=true , address=167.17.72.68:4656 , piece_id=X5TKPYMBYPKGTN5UPZLUH6YFTQD255FBZ3RRZPDIXT4BH7N5LH7A, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.512167554 (248h46m7.106278805s) [noise=true , address=45.91.116.7:4022 , piece_id=XGSY7J4JLGFBYNFD77TASFRHMUEOBXR5LV45NNG5HAJNXI2TRRZA, seg_pos={0 0}, upload=239169]: Uploading piece... Apr 18 01:02:56.598651932 (248h46m7.192763184s) [noise=true , address=145.79.227.32:4841 , piece_id=KLVYZYDXDVLNQXV7K2C3NGTYJ7DUZ5MNFWHETCMGXKHU7XXMPF6A, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:1FKz4ioR4bYX3jEHXQAUGeJwhCxzVtXfA3kddFrb84YNpJeRQD, address:145.79.227.32:4841): protocol: rpc: dial tcp 145.79.227.32:4841: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:02:57.104401325 (248h46m7.698512555s) [upload=239169]: Next returned nil segment Apr 18 01:02:57.104424156 (248h46m7.6985354s) [upload=239169]: Waiting for error group managing segments... Apr 18 01:02:57.164702344 (248h46m7.758813604s) [noise=true , address=45.91.116.13:4180 , piece_id=B43KJ5UDZZLHVP3ALBC4YR6CGE6VWEZTPU3TVOWTDHB7GNQNOX5Q, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: storage node overloaded, request limit: 2000 Apr 18 01:02:57.267764797 (248h46m7.86187604s) [noise=true , address=145.79.227.13:4836 , piece_id=6VLRGLPLLMHFCJL7SBCA3K6CRW2GLM2L52NAIS32772OQAIZGAAA, seg_pos={0 0}, upload=239169]: Done uploading piece. err: <nil> Apr 18 01:02:57.280927086 (248h46m7.875038316s) [noise=true , address=145.79.227.35:4496 , piece_id=UFHQUD5KS6SP5TDMSNKRBHY6UURWLIS5PRB2FM4NZ72ZFAKXRC5A, seg_pos={0 0}, upload=239169]: Done uploading piece. err: <nil> Apr 18 01:02:58.857759726 (248h46m9.451870966s) [noise=true , address=193.149.164.143:4697 , piece_id=MOGEYQYX4RH5SNIOWFT2FKBZPWHROP45GB6I2W2CES7QP6IVKTLQ, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, address:193.149.164.143:4697): protocol: storage node overloaded, request limit: 2000 Apr 18 01:03:02.316968680 (248h46m12.911079922s) [noise=true , address=45.91.116.7:4022 , piece_id=XGSY7J4JLGFBYNFD77TASFRHMUEOBXR5LV45NNG5HAJNXI2TRRZA, seg_pos={0 0}, upload=239169]: Done uploading piece. err: <nil> Apr 18 01:03:06.855954809 (248h46m17.45006605s) [noise=true , address=145.79.227.2:4120 , piece_id=D3KOTH3LRJWDGASDYKDVGB7GPKIYIZ43LIXYHCMENSD3F3YZ3UDA, seg_pos={0 0}, upload=239169]: Done uploading piece. err: <nil> Apr 18 01:03:56.094580656 (248h47m6.688691917s) [noise=true , address=167.17.72.68:4656 , piece_id=X5TKPYMBYPKGTN5UPZLUH6YFTQD255FBZ3RRZPDIXT4BH7N5LH7A, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:12bgJ2awZnNT2XAxjqB9xa5gkH9saVPiMLp6jRxvAmYeCipGvwe, address:167.17.72.68:4656): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:03:56.094658535 (248h47m6.688769785s) [noise=true , address=145.79.227.11:4751 , piece_id=B3MQUGOX5FHZ4GRIULVIVJVVG4HOAQXZHZQG5UX7LK2RW4S2J3HA, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:1rJkr2qSmcxGv7smeBBLwVjLoWz3Wsd7CvWWc3B7mzP5dBG3Uz, address:145.79.227.11:4751): protocol: rpc: dial tcp 145.79.227.11:4751: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:03:56.094871557 (248h47m6.688982805s) [noise=true , address=167.17.72.71:4803 , piece_id=TFTSTBRC6DXP6XFSATU6IPXVYXBKMI3Z276XNOGCRAP6PSKK2RAA, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:14byHdm6aw7qLgeiewQHaSowSte7meJcTdMiLUangnDmVTL3qR, address:167.17.72.71:4803): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:03:56.095015210 (248h47m6.689126467s) [noise=true , address=145.79.227.31:4816 , piece_id=AD66F4MD2ZWBVF7BAJEE3TZV63ZSXNIAV3NSDI3KNHN53O6N4MJA, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:1nq6pTeLCMEmtagczJkTMAdpADG5JrbSKhUjXAHy9oeS1pTpSx, address:145.79.227.31:4816): protocol: rpc: dial tcp 145.79.227.31:4816: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:03:56.095078947 (248h47m6.689190197s) [noise=true , address=167.17.72.70:4772 , piece_id=M5LFDCH57OKMYEYSMTZLBMMON6AHVMTXNLDDLFKRHKJ2FEW4HZ6Q, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:12PGtQc63J4cgYv18pkTeE639aFrf6mMp24xkWjynWnibe4QHFH, address:167.17.72.70:4772): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:03:56.095360204 (248h47m6.68947146s) [noise=true , address=167.17.72.69:4744 , piece_id=J2BH2OXEQBRJE4ZTHGWM3IGQHXWXDSJTS7XNBL5PVYXWTLF2XIWA, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:12jeqMW9gHwkjvGg1Z498L8FWvp6bpMbN5gdfoqqLR5X9UsQJoa, address:167.17.72.69:4744): protocol: rpc: dial tcp 167.17.72.69:4744: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:03:56.095390020 (248h47m6.689501271s) [noise=true , address=193.149.164.66:4990 , piece_id=NNSDWLKJD5MLSAP2ZDJPEUTR2ONG6TK64NSCSHOCGN5EYKF7YWGQ, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:12rZoTqAdgDDLWrSLga3M8CKSCNayUX33YFx5HB6g1FDjrVHUcq, address:193.149.164.66:4990): protocol: write tcp 139.59.144.22:55320->193.149.164.66:4990: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:03:56.095718528 (248h47m6.689829784s) [noise=true , address=145.79.227.44:4202 , piece_id=JU3VAO6347F2ID2ZXRHN6N3PQQ7VGOFFTGAWPUHOMEFTCNCQQG5A, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:145.79.227.44:4202): protocol: rpc: dial tcp 145.79.227.44:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:03:56.095879439 (248h47m6.689990687s) [noise=true , address=145.79.227.34:4915 , piece_id=U4P7JD2PA2BEBF7CHAOXYQXZPGYPQBXQDT6QKHBLU7ZS2XZI47DQ, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT, address:145.79.227.34:4915): protocol: rpc: dial tcp 145.79.227.34:4915: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:03:56.096343314 (248h47m6.690454567s) [noise=true , address=145.79.227.38:4010 , piece_id=5WRR2YTDS5ERLD36KBHLSZBVM6YPBOPFOHLRQV4XTDRWN3K7FZQQ, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:125Jkbj6bsEyRKAhrZPH1Hso9Eaou6Ly6qHKpKFqvK8giPbknjY, address:145.79.227.38:4010): protocol: rpc: dial tcp 145.79.227.38:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:03:56.096711972 (248h47m6.690823231s) [noise=true , address=145.79.227.29:4962 , piece_id=V44ZJREUTIQ6Y44SPIBYZKDDHJOPBWYG7TAKVSIDMJGLHLAOH2PA, seg_pos={0 0}, upload=239169]: Done uploading piece. err: ecclient: upload failed (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3, address:145.79.227.29:4962): protocol: rpc: dial tcp 145.79.227.29:4962: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:03:56.096752169 (248h47m6.6908634s) [seg_pos={0 0}, upload=239169]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Apr 18 01:03:56.203570287 (248h47m6.797681531s) [upload=239169]: Done uploading segments... Apr 18 01:03:56.203630804 (248h47m6.797742053s) [upload=239169]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.111083095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [......X....X..X..X..X....X....X.....X....XX..X..........XX..] duration: 5.697332512s trace: https://jaeger.dev.storj.io/trace/76d1b8f34010c80b FAIL (2026-04-18 01:03:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.697364484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..XX] duration: 5.001242217s trace: https://jaeger.dev.storj.io/trace/1deeb22ea55ce0df FAIL (2026-04-18 01:03:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001269477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.......X....X..X..X..X....X....X.....X....XX..X..........XX.] duration: 10.254798631s trace: https://jaeger.dev.storj.io/trace/76d1b8f34010c80b FAIL (2026-04-18 01:03:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.254852025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [........X....X..X..X..X....X....X.....X....XX..X..........XX] duration: 1m0.001169194s trace: https://jaeger.dev.storj.io/trace/76d1b8f34010c80b FAIL (2026-04-18 01:03:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001183044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 36 out of 60 is failed (threshold=3) [.......................XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.109368012s trace: https://jaeger.dev.storj.io/trace/5ae5abeca67f7601 FAIL (2026-04-18 01:02:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== 49.13.140:4513 , piece_id=BMHOCURPRG662ZTB6JKVXX2FRUFYRGD4LNDRO3VW75I2Z7YPOT4Q, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.230718259 (248h45m6.824829514s) [noise=true , address=167.17.72.71:4804 , piece_id=EJNJMTGQLITEMZ3AMIX5JKMUL5PACAWEKAX54QAIBVX2OHWC46KQ, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.231108463 (248h45m6.825219714s) [noise=true , address=145.79.227.29:4962 , piece_id=GL2CAMZTNTIEN5UIETZQUVRUXXGJQZ7XESI2HS6JCU2VBGZXAOEA, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.231715601 (248h45m6.825826846s) [noise=true , address=145.79.227.50:4382 , piece_id=ZMP54O3ZBNICJNQRP6EG3IMQXNQSH2FWP6MSTCXTP6PJE5BGJGVQ, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.232383510 (248h45m6.826494763s) [noise=true , address=193.149.164.143:4697 , piece_id=VXXNIQBCZLF7I4XHYCWNU7WUAXS63CEDLH3QLDNYK5CT2EFROBIQ, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.232985479 (248h45m6.827096727s) [noise=true , address=193.149.164.71:4314 , piece_id=GTPEHJFTCUOGXM24OY4QEML74PLIRQ6A6CYKFHTREB2XPQUI5R3A, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.233559695 (248h45m6.827670945s) [noise=true , address=45.91.116.13:4180 , piece_id=ZNDHC5TN3IWDDCRQB365KM7ZP2PA3VWNZWORTDGRTSFDTH3F5KZA, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.234134559 (248h45m6.82824581s) [noise=true , address=193.149.164.66:4965 , piece_id=B4AIFWCYLCWPWCOYBETE524L5BPWND5UTXMA47YH5Y6P4OGBGIGA, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.234285537 (248h45m6.828396783s) [noise=true , address=145.79.227.33:4903 , piece_id=RLL3AXURC3Q333BPFZEHLIFKAJJ7EA7XWUJRR3FIJI6E5GA3FX5Q, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.234788990 (248h45m6.828900237s) [noise=true , address=167.17.72.68:4645 , piece_id=HOYOTG3JK7EGAR6DU5FSMJX6FH74LAGWJJMETHCVHNVZ4K2J3DQA, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.235279727 (248h45m6.829390976s) [noise=true , address=167.17.72.70:4755 , piece_id=UK2U72BYR3G66VTB3Q44TMC4YXNQJSGGUAYNZSYY3WL47N25AWQQ, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.235729077 (248h45m6.829840324s) [noise=true , address=167.17.72.69:4720 , piece_id=V7R3SD7EN3B4QGUSRWP5ISAPSMLHCDU7IGSCMZ65IDAOXNF6PKXA, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.236159855 (248h45m6.830271105s) [noise=true , address=145.79.227.13:4808 , piece_id=2UBSJDQ2G64YFK247FNVIVXB3OTEG2OH4GZS33T2W6ETGUMOFW7A, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.236786277 (248h45m6.830897523s) [noise=true , address=145.79.227.11:4754 , piece_id=PON5BS537N4RZDAET2WLA2HL7R675NAZQODJWGE4KWMZ2KCZX7OA, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.237296876 (248h45m6.831408122s) [noise=true , address=145.79.227.49:4359 , piece_id=3Y6P7EGQVSUMJ4NAG3STNBPTXL7T2GQDWWX5VMRTGHPRNMOC2ITQ, seg_pos={0 0}, upload=239152]: Uploading piece... Apr 18 01:01:56.326258975 (248h45m6.920370224s) [noise=true , address=145.79.227.49:4359 , piece_id=3Y6P7EGQVSUMJ4NAG3STNBPTXL7T2GQDWWX5VMRTGHPRNMOC2ITQ, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV, address:145.79.227.49:4359): protocol: rpc: dial tcp 145.79.227.49:4359: connect: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:01:56.798822531 (248h45m7.392933763s) [upload=239152]: Next returned nil segment Apr 18 01:01:56.798871762 (248h45m7.392983001s) [upload=239152]: Waiting for error group managing segments... Apr 18 01:01:56.808740657 (248h45m7.40285191s) [noise=true , address=45.91.116.13:4180 , piece_id=ZNDHC5TN3IWDDCRQB365KM7ZP2PA3VWNZWORTDGRTSFDTH3F5KZA, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:1zPViJ6hkDLdzVwYruwkqYjvEfFHuDXNcKyx6ogPCPec8dvetG, address:45.91.116.13:4180): protocol: storage node overloaded, request limit: 2000 Apr 18 01:01:56.911066693 (248h45m7.505177932s) [noise=true , address=193.149.164.143:4697 , piece_id=VXXNIQBCZLF7I4XHYCWNU7WUAXS63CEDLH3QLDNYK5CT2EFROBIQ, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, address:193.149.164.143:4697): protocol: storage node overloaded, request limit: 2000 Apr 18 01:01:56.983924971 (248h45m7.578036232s) [noise=true , address=145.79.227.13:4808 , piece_id=2UBSJDQ2G64YFK247FNVIVXB3OTEG2OH4GZS33T2W6ETGUMOFW7A, seg_pos={0 0}, upload=239152]: Done uploading piece. err: <nil> Apr 18 01:01:56.995282193 (248h45m7.589393442s) [noise=true , address=145.79.227.32:4866 , piece_id=JVZRELIDXHBIC6LXRTAZQK3CX6OU4V35NPTUE5QIMFR7THFXZZRA, seg_pos={0 0}, upload=239152]: Done uploading piece. err: <nil> Apr 18 01:01:57.017485518 (248h45m7.611596767s) [noise=true , address=145.79.227.50:4382 , piece_id=ZMP54O3ZBNICJNQRP6EG3IMQXNQSH2FWP6MSTCXTP6PJE5BGJGVQ, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:1SN5izFGjg4t9qkUVEwF71nNgK2Fjpzo3wuZPJae5qc6QLN8Nf, address:145.79.227.50:4382): protocol: storage node overloaded, request limit: 2000 Apr 18 01:01:57.514345176 (248h45m8.108456415s) [noise=true , address=193.149.164.71:4314 , piece_id=GTPEHJFTCUOGXM24OY4QEML74PLIRQ6A6CYKFHTREB2XPQUI5R3A, seg_pos={0 0}, upload=239152]: Done uploading piece. err: <nil> Apr 18 01:01:57.755034999 (248h45m8.349146256s) [noise=true , address=193.149.164.66:4965 , piece_id=B4AIFWCYLCWPWCOYBETE524L5BPWND5UTXMA47YH5Y6P4OGBGIGA, seg_pos={0 0}, upload=239152]: Done uploading piece. err: <nil> Apr 18 01:01:58.849172733 (248h45m9.443283983s) [noise=true , address=64.49.13.140:4513 , piece_id=BMHOCURPRG662ZTB6JKVXX2FRUFYRGD4LNDRO3VW75I2Z7YPOT4Q, seg_pos={0 0}, upload=239152]: Done uploading piece. err: <nil> Apr 18 01:01:58.934442174 (248h45m9.528553416s) [noise=true , address=77.90.31.19:4860 , piece_id=VYGNFVJBWSEMMYIXBRX6QPVXDFSKOW24LCUYN24II3CPHQCBZ3PA, seg_pos={0 0}, upload=239152]: Done uploading piece. err: <nil> Apr 18 01:02:55.831073901 (248h46m6.425185153s) [noise=true , address=167.17.72.69:4720 , piece_id=V7R3SD7EN3B4QGUSRWP5ISAPSMLHCDU7IGSCMZ65IDAOXNF6PKXA, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:12evKRTeCTTPWorcqVfLQ5vbg1kZvcB2atkURCXB5dq4VSfrKJD, address:167.17.72.69:4720): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:02:55.831145680 (248h46m6.42525692s) [noise=true , address=167.17.72.68:4645 , piece_id=HOYOTG3JK7EGAR6DU5FSMJX6FH74LAGWJJMETHCVHNVZ4K2J3DQA, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:12LkJ3iu6FoMMpqkwtLrD9Q5zTrwk7s57bPtJzRw1kgzXTih6kE, address:167.17.72.68:4645): protocol: rpc: dial tcp 167.17.72.68:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:02:55.831393554 (248h46m6.425504805s) [noise=true , address=145.79.227.11:4754 , piece_id=PON5BS537N4RZDAET2WLA2HL7R675NAZQODJWGE4KWMZ2KCZX7OA, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:125aPwoHeFabk6zZ6bkGNj3USJYgTQG3RB1QhhXssj7Vik2AuCs, address:145.79.227.11:4754): protocol: rpc: dial tcp 145.79.227.11:4754: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:02:55.831624031 (248h46m6.425735278s) [noise=true , address=145.79.227.29:4962 , piece_id=GL2CAMZTNTIEN5UIETZQUVRUXXGJQZ7XESI2HS6JCU2VBGZXAOEA, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:13va7LjDtQV2ob7mrUaJUXc64HmZG3svoDaErmypjYHVAcw1o3, address:145.79.227.29:4962): protocol: rpc: dial tcp 145.79.227.29:4962: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:02:55.831722419 (248h46m6.42583367s) [noise=true , address=145.79.227.34:4936 , piece_id=TI66WGYTLWALPTXMGZFK6WNN32GHZEGRDFFV2HNRNOTNEGMNZCYA, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:145.79.227.34:4936): protocol: rpc: dial tcp 145.79.227.34:4936: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:02:55.831961875 (248h46m6.426073123s) [noise=true , address=145.79.227.33:4903 , piece_id=RLL3AXURC3Q333BPFZEHLIFKAJJ7EA7XWUJRR3FIJI6E5GA3FX5Q, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:12mY4YcLLVbnVuXtKSHwyv4sgTycqzxUZHGVQha5WfXXTPjNGLq, address:145.79.227.33:4903): protocol: rpc: dial tcp 145.79.227.33:4903: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:02:55.832074995 (248h46m6.426186246s) [noise=true , address=167.17.72.70:4755 , piece_id=UK2U72BYR3G66VTB3Q44TMC4YXNQJSGGUAYNZSYY3WL47N25AWQQ, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:12UEB7jdcocUd24j1gEMoWYYPvWzn9q4oBwCxJmMnAW18WEDA9r, address:167.17.72.70:4755): protocol: expected piece hash; context deadline exceeded; EOF Apr 18 01:02:55.832475281 (248h46m6.42658653s) [noise=true , address=167.17.72.71:4804 , piece_id=EJNJMTGQLITEMZ3AMIX5JKMUL5PACAWEKAX54QAIBVX2OHWC46KQ, seg_pos={0 0}, upload=239152]: Done uploading piece. err: ecclient: upload failed (node:1LTfwZPCXf6dCYUGyRZwE9fC4aj3w6HA41iBmntg4G5AW6iCY2, address:167.17.72.71:4804): protocol: write tcp 139.59.144.22:43392->167.17.72.71:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 18 01:02:55.832514956 (248h46m6.426626199s) [seg_pos={0 0}, upload=239152]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Apr 18 01:02:55.935137998 (248h46m6.52924926s) [upload=239152]: Done uploading segments... Apr 18 01:02:55.935191733 (248h46m6.529302983s) [upload=239152]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.109389961s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [.X.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXXXXXX.XXXX..X] duration: 5.000855828s trace: https://jaeger.dev.storj.io/trace/500ab6051831cd81 FAIL (2026-04-18 01:02:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000902747s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)