Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.603427812s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:40:01 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.603453618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 3.154680723s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:39:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.154699433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.937398073s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:39:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.937424601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 2.03191351s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:39:24 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.031944394s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 1.584085311s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:38:53 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.584166666s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.949960367s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:38:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.949986196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 2.151224492s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:38:28 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.151251125s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 1.711789403s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:38:05 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.711831096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.699484189s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:37:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.699545698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 3.221333184s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:36:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.221425196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 1.608236589s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:36:58 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.608284414s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.964243921s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:36:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.964266971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 1.685830896s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:36:09 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.685919523s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 1.634428685s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:35:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.634456771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.71687955s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:35:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.71690787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 1.673818793s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:35:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.67394632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 2.174127897s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:34:47 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.174157538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 3.021376203s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:34:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.021405476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 1.736008547s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:33:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.736029165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 3.335421352s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:33:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.335448337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 1.58167369s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:33:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.581726118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.914156041s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:32:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.914180217s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 2.006765227s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:32:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.006795498s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 3.059499658s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:31:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.059530649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 2.226416583s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:31:42 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.226465311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.821389673s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:31:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.821482762s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 2.368695201s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:31:06 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.368803745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 1.568652174s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:30:42 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.56871602s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.664279485s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:30:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.664344604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 1.795877528s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:30:09 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.79590073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 3.250200879s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:30:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.250236865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.964761491s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:29:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.964789294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 1.65736038s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:29:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.657381597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 35.149065029s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:28:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 35.149093859s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 2.182194781s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:28:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.182576801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 1.695383704s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:28:01 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.695411017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.699777577s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:27:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.699807514s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 2.202226212s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:27:11 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.20224267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.980067231s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:26:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.980096035s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 3.046274971s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:26:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.046306413s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 2.159858942s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:26:05 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.159952404s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 1.679877977s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:25:17 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.679902155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.856599638s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:25:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.856625959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.468027498s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:24:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.46804205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 2.07047012s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:24:21 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.070742125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.569057704s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:23:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.56916394s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 1.906809073s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:23:27 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.906830322s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 1.765646781s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:22:55 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.765678784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.057480796s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:22:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.057586391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 1.754637988s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:21:40 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.754651684s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)