Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration: 2.71750502s FAIL (2025-09-23 19:07:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.717583345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.457386626s FAIL (2025-09-23 19:07:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.457481226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.635734271s FAIL (2025-09-23 19:07:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.635764186s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration: 2.733832103s FAIL (2025-09-23 19:06:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.733863196s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.660131897s FAIL (2025-09-23 19:06:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.660178584s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.488919291s FAIL (2025-09-23 19:05:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.488952706s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XXXX.....................] duration: 2.932880571s FAIL (2025-09-23 19:05:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.932905696s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 2.718557009s FAIL (2025-09-23 19:05:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.718833395s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.512105089s FAIL (2025-09-23 19:05:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.512141247s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 2.660020457s FAIL (2025-09-23 19:04:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.660041752s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.51486828s FAIL (2025-09-23 19:04:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.514905957s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 2.922450378s FAIL (2025-09-23 19:04:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.92248588s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.649663861s FAIL (2025-09-23 19:03:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.649701524s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 2.701766497s FAIL (2025-09-23 19:03:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.701792368s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 2.752851348s FAIL (2025-09-23 19:02:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.752876345s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.749949732s FAIL (2025-09-23 19:02:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.749977647s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XXXX...............] duration: 2.900458789s FAIL (2025-09-23 19:01:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.900516278s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.669904229s FAIL (2025-09-23 19:01:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.66995057s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration: 2.907489529s FAIL (2025-09-23 19:00:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.907534864s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.540399844s FAIL (2025-09-23 19:00:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.540429189s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration: 2.830304362s FAIL (2025-09-23 19:00:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.830327528s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.348962047s FAIL (2025-09-23 18:59:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.348992508s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XXXX............] duration: 2.766744696s FAIL (2025-09-23 18:59:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.766833043s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.449746999s FAIL (2025-09-23 18:58:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.449780446s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XXXX...........] duration: 2.649894819s FAIL (2025-09-23 18:58:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.649915874s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XXXX..........] duration: 2.88747619s FAIL (2025-09-23 18:58:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.887576354s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.59751024s FAIL (2025-09-23 18:57:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.597542188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.513757696s FAIL (2025-09-23 18:57:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.513791917s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XXXX.........] duration: 2.823116039s FAIL (2025-09-23 18:57:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.82313623s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................XXXX........] duration: 2.93866274s FAIL (2025-09-23 18:56:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.938686451s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.354606781s FAIL (2025-09-23 18:55:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.35467257s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XXXX.......] duration: 2.781157002s FAIL (2025-09-23 18:55:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.781192906s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.490441951s FAIL (2025-09-23 18:55:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.490481105s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XXXX......] duration: 2.891085954s FAIL (2025-09-23 18:55:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.891108652s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.567278321s FAIL (2025-09-23 18:54:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.567305301s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XXXX.....] duration: 2.892512348s FAIL (2025-09-23 18:54:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.89253573s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.412486475s FAIL (2025-09-23 18:54:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.41253269s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XXXX....] duration: 2.928198447s FAIL (2025-09-23 18:53:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.928252124s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.348156408s FAIL (2025-09-23 18:53:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.348188486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.372472373s FAIL (2025-09-23 18:53:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.372507357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.472817656s FAIL (2025-09-23 18:53:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.47284619s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXXX...] duration: 2.928347734s FAIL (2025-09-23 18:53:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.928401533s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration: 2.681755123s FAIL (2025-09-23 18:52:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.681856433s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.391674087s FAIL (2025-09-23 18:52:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.391706646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.351768457s FAIL (2025-09-23 18:51:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.351807573s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXXX.] duration: 2.904301633s FAIL (2025-09-23 18:51:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.904334927s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.672631995s FAIL (2025-09-23 18:51:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.672661057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.403119342s FAIL (2025-09-23 18:50:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.40313938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 18.309667094s FAIL (2025-09-23 18:50:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 18.309690032s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.000540911s FAIL (2025-09-23 18:50:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000557412s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)