Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.485526654s FAIL (2025-09-23 19:25:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.485567509s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XXXX.................................................] duration: 2.730701035s FAIL (2025-09-23 19:25:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.730754554s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.688347951s FAIL (2025-09-23 19:25:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.688384652s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XXXX................................................] duration: 2.842066837s FAIL (2025-09-23 19:25:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.842103331s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XXXX...............................................] duration: 2.798656049s FAIL (2025-09-23 19:24:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.800051056s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.508493044s FAIL (2025-09-23 19:24:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.508520365s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XXXX..............................................] duration: 2.701799778s FAIL (2025-09-23 19:23:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.701825206s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.443844519s FAIL (2025-09-23 19:23:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.443882577s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XXXX.............................................] duration: 2.709594555s FAIL (2025-09-23 19:23:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.709862976s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XXXX............................................] duration: 2.878273767s FAIL (2025-09-23 19:22:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.878590291s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XXXX...........................................] duration: 2.70709741s FAIL (2025-09-23 19:22:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.707121951s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.409798265s FAIL (2025-09-23 19:22:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.409833632s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XXXX..........................................] duration: 2.636342747s FAIL (2025-09-23 19:21:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.636965767s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.502856037s FAIL (2025-09-23 19:21:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.502895554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.415354876s FAIL (2025-09-23 19:21:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.415399056s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XXXX.........................................] duration: 2.930963669s FAIL (2025-09-23 19:20:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.930993899s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.420923914s FAIL (2025-09-23 19:20:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.420956109s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XXXX........................................] duration: 2.803419729s FAIL (2025-09-23 19:20:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.803445622s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.56535105s FAIL (2025-09-23 19:20:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.56538324s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XXXX.......................................] duration: 2.743868024s FAIL (2025-09-23 19:19:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.743956655s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.503243228s FAIL (2025-09-23 19:19:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.503264235s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XXXX......................................] duration: 2.945429826s FAIL (2025-09-23 19:19:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.945456071s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.440054038s FAIL (2025-09-23 19:19:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.440085914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.475627173s FAIL (2025-09-23 19:18:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.475684167s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XXXX.....................................] duration: 2.912876601s FAIL (2025-09-23 19:18:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.912993607s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.463386043s FAIL (2025-09-23 19:17:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.463413066s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XXXX....................................] duration: 2.760708848s FAIL (2025-09-23 19:17:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.760735971s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.474019149s FAIL (2025-09-23 19:16:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.474053983s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XXXX...................................] duration: 2.865775936s FAIL (2025-09-23 19:16:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.865854324s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.512484192s FAIL (2025-09-23 19:16:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.512542099s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XXXX..................................] duration: 2.612036349s FAIL (2025-09-23 19:16:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.61209473s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.390237421s FAIL (2025-09-23 19:15:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.390264465s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XXXX.................................] duration: 2.861307308s FAIL (2025-09-23 19:15:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.861357651s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.570089671s FAIL (2025-09-23 19:14:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.570116151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.353463469s FAIL (2025-09-23 19:14:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.35349598s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XXXX................................] duration: 2.751473339s FAIL (2025-09-23 19:14:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.751496099s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................XXXX...............................] duration: 2.977059261s FAIL (2025-09-23 19:13:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.977100885s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.5816315s FAIL (2025-09-23 19:12:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.581666397s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................XXXX..............................] duration: 2.730122126s FAIL (2025-09-23 19:12:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.730144722s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.386402338s FAIL (2025-09-23 19:12:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.386427918s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................XXXX.............................] duration: 2.828834736s FAIL (2025-09-23 19:11:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.828874393s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................XXXX............................] duration: 2.738328203s FAIL (2025-09-23 19:11:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.738359597s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.601162768s FAIL (2025-09-23 19:11:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.601199325s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration: 2.913533275s FAIL (2025-09-23 19:10:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.913560688s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.382949381s FAIL (2025-09-23 19:10:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.382977985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.472982817s FAIL (2025-09-23 19:09:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.47300968s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................XXXX..........................] duration: 3.088857656s FAIL (2025-09-23 19:09:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.088882949s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................XXXX.........................] duration: 2.855106787s FAIL (2025-09-23 19:09:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.855131141s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XXXX........................] duration: 2.658811703s FAIL (2025-09-23 19:08:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.65890657s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.473336349s FAIL (2025-09-23 19:08:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 23 18:48:27.071761249 (5h18m14.87522583s) [upload=4632]: Starting upload Sep 23 18:48:27.071844498 (5h18m14.875309093s) [upload=4632]: Done starting upload Sep 23 18:48:27.071922976 (5h18m14.875387554s) [upload=4632]: Uploading segments... Sep 23 18:48:27.072100574 (5h18m14.875565167s) [upload=4632]: Got next segment. Inline: false Sep 23 18:48:27.072169755 (5h18m14.875634335s) [upload=4632]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 23 18:48:27.072175756 (5h18m14.875640347s) [upload=4632]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 23 18:48:27.252485055 (5h18m15.055949656s) [seg_pos={0 0}, upload=4632]: Begin upload segment... Sep 23 18:48:27.252544796 (5h18m15.056009395s) [seg_pos={0 0}, upload=4632]: Done begin upload segment. Sep 23 18:48:27.252574818 (5h18m15.056039422s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.259786732 (5h18m15.063251334s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.261424618 (5h18m15.064889219s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.263205946 (5h18m15.066670542s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.264985460 (5h18m15.068450057s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.266786187 (5h18m15.070250891s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.268394959 (5h18m15.071859716s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.269229524 (5h18m15.072694123s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.270092520 (5h18m15.07355712s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.271066258 (5h18m15.074530853s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.272668000 (5h18m15.076132603s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.273266904 (5h18m15.076731504s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275204173 (5h18m15.078668775s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.275908609 (5h18m15.079373205s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277231628 (5h18m15.080696205s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.277884907 (5h18m15.081349502s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Uploading piece... Sep 23 18:48:27.881885858 (5h18m15.685350434s) [upload=4632]: Next returned nil segment Sep 23 18:48:27.881905136 (5h18m15.685369726s) [upload=4632]: Waiting for error group managing segments... Sep 23 18:48:28.016523029 (5h18m15.819987622s) [noise=true , address=45.91.116.14:4189 , piece_id=7Z7GT4GYOJKD2NVOD5EKBTRYOLU56EVJ7OKZVMM6K6ZMEP367IPQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.017292495 (5h18m15.820757079s) [noise=true , address=45.91.116.6:4975 , piece_id=6OVX6X3KIPEHF5RGOGCJKB2H4AM4LJRROLPC6AQN5FZ5JB34YAQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.018428684 (5h18m15.821893282s) [noise=true , address=45.91.116.5:4298 , piece_id=KXNGXZSYDYEXSZ5T5KJLKC6PIQYWJXG7RGGAR6J3NXLICTNXHXVA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019532942 (5h18m15.822997538s) [noise=true , address=45.91.116.4:4939 , piece_id=NQOZP6NJ4AHBAZAUXRBKJRULEFIIKOVAQC4CJF4RCOEBZWWRU2CA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.019840213 (5h18m15.823304813s) [noise=true , address=45.91.116.12:4225 , piece_id=YWPRV3UHCYZ7ZK6FB2FAOE6BU3DE4QHF5DJADO3LVTASCXAL425Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.029655046 (5h18m15.833119627s) [noise=true , address=45.91.116.8:4047 , piece_id=BCK33CBO6B3GWUNEDSYDJLZCF5CLRAWFH5JZ6UOEHV3GOSMEPLQA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.031565646 (5h18m15.835030238s) [noise=true , address=77.90.31.16:4734 , piece_id=63QQF7XI6T5KX7XB5MSEVS4ZY4ZZ5WDKWLGMJL3YUO4URB4N3MKQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.038750186 (5h18m15.842214791s) [noise=true , address=77.90.31.10:4520 , piece_id=WNUH6FGATWV2WPRW7DVLF3UPYP5PGGBLEPYITMEP45PYQPXGPBOQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:28.052415828 (5h18m15.855880418s) [noise=true , address=77.90.31.17:4771 , piece_id=MWME5UAEW2KC2PYZR54SNHO6EFWSB35SXAZRSTSANHRUKOFQ3F2Q, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:40.435844938 (5h18m28.23930953s) [noise=true , address=145.79.227.15:4875 , piece_id=6KZBL6OHCEMWRCXUS7J2HEOACCINFH26AWOIABOIOABFLQL2UTEQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:53.946852052 (5h18m41.750316644s) [noise=true , address=145.79.227.11:4732 , piece_id=4Q462223V752OAQY5H5O4K4TBYACIERWKS3YPOV6GMY2JDR42ORQ, seg_pos={0 0}, upload=4632]: Done uploading piece. err: <nil> Sep 23 18:48:56.762365912 (5h18m44.565830506s) [noise=true , address=145.79.227.13:4804 , piece_id=2GOLBVFQ7BMGPRS52CVO6R346FPSZCFV7K5WDWKARRPC4AKOMZXA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762470617 (5h18m44.565935202s) [noise=true , address=145.79.227.18:4007 , piece_id=SVNRCB6DJHAWINQHQWNIXTZP4YBHEZT4OTQ4SU47NBTMP5HFZHNA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:145.79.227.18:4007): protocol: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53046->145.79.227.18:4007: use of closed network connection Sep 23 18:48:56.762497707 (5h18m44.565962299s) [noise=true , address=145.79.227.7:4295 , piece_id=UUH3PVGX76DSR2WPLF5IASLQPX745DRG7XTTSEOJMBVF25CEB5OA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:145.79.227.7:4295): protocol: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56488->145.79.227.7:4295: use of closed network connection Sep 23 18:48:56.762569129 (5h18m44.566033721s) [noise=true , address=145.79.227.12:4767 , piece_id=5UIPNF6WPPEYSBFI2NLE4MWC6W2F6IP3MYUQSET3UQFK25NG4K6A, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762634698 (5h18m44.566099291s) [noise=true , address=145.79.227.16:4910 , piece_id=KE4O64FTIHRT53TOUL5A2OQ7JFQZGSM6GM7K5T65NW24N4PEP7EA, seg_pos={0 0}, upload=4632]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: expected piece hash; context deadline exceeded; EOF Sep 23 18:48:56.762660559 (5h18m44.566125152s) [seg_pos={0 0}, upload=4632]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 23 18:48:56.880060680 (5h18m44.683525262s) [upload=4632]: Done uploading segments... Sep 23 18:48:56.880087013 (5h18m44.683551605s) [upload=4632]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.473431131s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)