Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 1.723982145s trace: https://jaeger.dev.storj.io/trace/5f0196cb3fe1acaf FAIL (2026-03-07 00:28:31 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.724075069s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 1.605769906s trace: https://jaeger.dev.storj.io/trace/5f0196cb3fe1acaf FAIL (2026-03-07 00:27:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.605800977s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 1.946886684s trace: https://jaeger.dev.storj.io/trace/5f0196cb3fe1acaf FAIL (2026-03-07 00:27:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.946911267s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 1.657796378s trace: https://jaeger.dev.storj.io/trace/5f0196cb3fe1acaf FAIL (2026-03-07 00:26:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.65782012s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 2.219356514s trace: https://jaeger.dev.storj.io/trace/5f0196cb3fe1acaf FAIL (2026-03-07 00:26:05 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.219400716s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 1.676749559s trace: https://jaeger.dev.storj.io/trace/5f0196cb3fe1acaf FAIL (2026-03-07 00:25:10 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.676786704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001062733s trace: https://jaeger.dev.storj.io/trace/5f0196cb3fe1acaf FAIL (2026-03-07 00:24:30 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 5.001087288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 3.023479351s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 22:02:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.023508132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.081752605s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 22:01:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.081804396s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.142551922s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 22:00:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.142845212s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.871829811s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 22:00:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.871853519s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.979006313s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:59:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.979043744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.942022034s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:59:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.942052958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.893920078s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:58:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.893944087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.910248833s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:57:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.910273017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.783288652s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:56:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.783304747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.044599096s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:55:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.04463169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.678520462s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:54:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.678550066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 3.0118826s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:54:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.011916423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.097200269s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:52:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.097235962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 3.25401431s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:51:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.254104367s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 3.028272401s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:51:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.028297829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 3.021587457s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:50:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.021618642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.08864361s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:49:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.088668568s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.656821677s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:49:20 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.656859752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.854818569s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:49:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.854845455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.649857232s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:49:03 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.649888323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.967626097s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:48:18 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.967739146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.620743822s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:48:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.620775052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.787168489s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:47:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.787208385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 2.156920566s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:47:15 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.156940287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.94894131s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:46:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.948969612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 2.272734422s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:46:15 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.272776797s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.870450466s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:46:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.870477099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.746476538s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:45:53 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.746567376s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.700551779s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:45:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.70057924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.025075688s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:45:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.02510192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 3.105105568s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:44:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.105140288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.664154705s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:44:38 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.664192803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 2.354235907s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:44:10 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.35429011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.958877956s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:43:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.958904772s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.768467404s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:43:25 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.768493162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.933673651s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:43:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.933706516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 2.152296994s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:42:39 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.152327864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.27033662s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:42:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.270359557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 2.171558615s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:42:14 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.171620516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.774980922s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:41:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.775010386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.78020343s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:41:34 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.780227819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.119118057s trace: https://jaeger.dev.storj.io/trace/404143aab446e5b4 FAIL (2026-03-06 21:41:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 845s) [upload=9852]: Starting upload Mar 6 21:16:50.991153047 (10h16m10.803644998s) [upload=9852]: Done starting upload Mar 6 21:16:50.991246480 (10h16m10.803738424s) [upload=9852]: Uploading segments... Mar 6 21:16:50.991601308 (10h16m10.804093238s) [upload=9852]: Got next segment. Inline: false Mar 6 21:16:50.991679396 (10h16m10.804171346s) [upload=9852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 6 21:16:50.991688916 (10h16m10.804180871s) [upload=9852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 6 21:16:51.108515372 (10h16m10.921007348s) [seg_pos={0 0}, upload=9852]: Begin upload segment... Mar 6 21:16:51.108591176 (10h16m10.921083105s) [seg_pos={0 0}, upload=9852]: Done begin upload segment. Mar 6 21:16:51.108657133 (10h16m10.921149087s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109170181 (10h16m10.921662134s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.109567071 (10h16m10.922059023s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110100989 (10h16m10.922592942s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.110966263 (10h16m10.923458216s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.111212959 (10h16m10.923704908s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112194083 (10h16m10.924686033s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.112382000 (10h16m10.924873952s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113240885 (10h16m10.925732837s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.113611959 (10h16m10.926103917s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114190914 (10h16m10.926682866s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114544586 (10h16m10.927036538s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.114563816 (10h16m10.927055766s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115198489 (10h16m10.927690435s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.115599829 (10h16m10.928091781s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116273713 (10h16m10.928765663s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.116636228 (10h16m10.929128174s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.117235343 (10h16m10.929727293s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Uploading piece... Mar 6 21:16:51.664467158 (10h16m11.476959088s) [upload=9852]: Next returned nil segment Mar 6 21:16:51.664479542 (10h16m11.476971486s) [upload=9852]: Waiting for error group managing segments... Mar 6 21:16:51.826449037 (10h16m11.63894098s) [noise=true , address=45.91.116.14:4202 , piece_id=3SKSAWRP5J3EFBDQWQCQWA6CZJC4JYSCNZ6N5XUHCODSRVGAIY5Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.828749659 (10h16m11.641241602s) [noise=true , address=145.79.227.43:4179 , piece_id=RLDVHGHDIBSZ76DUIX4MJ2FL6O4TGGMJZ5EBD4YOL6WXMDUEBOGA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.829507559 (10h16m11.641999503s) [noise=true , address=45.91.116.8:4055 , piece_id=JLA6AGODXAIXX3NUNRGUY5DQ7EKRL3ITJJW5ALRB6YGXCC2P77VQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.840820640 (10h16m11.653312583s) [noise=true , address=145.79.227.16:4940 , piece_id=3UW7NN7BYSMW7SVASYQ2DRURWPNIDDLS2YWO4AGIHTQYIG23FGBA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.848876765 (10h16m11.661368708s) [noise=true , address=145.79.227.52:4479 , piece_id=DCLPQF3CQJFJI5HO6M3ZH5P36HFJJRXMA4YNIXVW2F4VPLV5MH6A, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.856460098 (10h16m11.668952045s) [noise=true , address=145.79.227.26:4667 , piece_id=KUKSBGBH4ZB4M7SL6RSK76M7DSOMHOXKHQ4R73ILMDCAH3LH6BEA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:51.885532591 (10h16m11.698024535s) [noise=true , address=45.91.116.6:4993 , piece_id=IIWNLZ2J2U2LPH5KXFVQFJ7OHEGPG7S34FTTYYLXIZADWEPEM6XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.150822529 (10h16m11.963314458s) [noise=true , address=167.17.72.68:4666 , piece_id=O4FJGIABYWPPHCRDWGXNDZPSOXGGKZVUBEZBCDXWW4QCIRRBCW4Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.214322374 (10h16m12.026814321s) [noise=true , address=167.17.72.70:4767 , piece_id=B4IOMSIS3YG2ZT3K6HU54FE3544Q4BZ7R4W6Q64DLZ2TIDOUA4MA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.220067686 (10h16m12.032559629s) [noise=true , address=167.17.72.69:4725 , piece_id=CL4E2FQHPO6PAA56A4ZH5XUQZOZMAFT52BVG6GPERASKT4GM6OEQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.249337015 (10h16m12.061828959s) [noise=true , address=167.17.72.71:4784 , piece_id=W43R3RDQD46P6V5M3PIZ54AOEVDW7XKKDGPLMUYDT5YSU4SQSNNA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.356649609 (10h16m12.169141554s) [noise=true , address=193.149.164.69:4086 , piece_id=2LMRBRJQMNSEGERS6Q3IAF2BCJ2EI2PSEI2HR56SWG6MAUF7N5SQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:16:52.358833287 (10h16m12.171325235s) [noise=true , address=193.149.164.67:4025 , piece_id=UPPHU7DWEP3OB476VJXMHOGOB6UDUDFJOHJV23H7QTE255Y4B6GA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: <nil> Mar 6 21:17:50.708031311 (10h17m10.520523262s) [noise=true , address=77.90.31.14:4689 , piece_id=HLGGNZCJSKT6GUUMVT3C5ALOMBR3EOYPMTQM3HL32B6LEZAT3RAA, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, address:77.90.31.14:4689): protocol: rpc: dial tcp 77.90.31.14:4689: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708301124 (10h17m10.520793076s) [noise=true , address=77.90.31.18:4809 , piece_id=FP7UISQF4JELGC6TA5DHPNN5D4K7PARW5J5D5YSZJFBE27YRX34Q, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1ZkWnew57Zj3H7XoUZHPo57Rzfw5awa7DYwuc7chcD4bJ8aGPz, address:77.90.31.18:4809): protocol: rpc: dial tcp 77.90.31.18:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708491230 (10h17m10.520983182s) [noise=true , address=77.90.31.15:4713 , piece_id=E7LV3A6O3YWEDVT7KHOQT7Z6B2QOXNVYP3QX5C5IYJR4N7JP4ZDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: rpc: dial tcp 77.90.31.15:4713: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.708636371 (10h17m10.521128316s) [noise=true , address=77.90.31.11:4586 , piece_id=EZNOWVK5KSW25RQPHWKSQDK5TW5HB2WOOHYDYD2ECR34DZWTS2XQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:12cp4U45mdzouvz4Gq1YPQ1EzuHttggJnXNepr1ai218HqkxeF2, address:77.90.31.11:4586): protocol: write tcp 139.59.144.22:60492->77.90.31.11:4586: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 6 21:17:50.709399331 (10h17m10.521891295s) [noise=true , address=77.90.31.19:4869 , piece_id=W6IWOTC75TVS74M6US7TRAO2BCKVQZKCINJNO4UFL6PTNB3CLVDQ, seg_pos={0 0}, upload=9852]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: context deadline exceeded; context deadline exceeded Mar 6 21:17:50.709464847 (10h17m10.521956783s) [seg_pos={0 0}, upload=9852]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 6 21:17:50.814029140 (10h17m10.626521074s) [upload=9852]: Done uploading segments... Mar 6 21:17:50.814066754 (10h17m10.626558698s) [upload=9852]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.119166455s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.583532637s trace: https://jaeger.dev.storj.io/trace/40416561fa90c3e6 FAIL (2026-03-06 21:40:54 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.583565815s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)